[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:08 dist-test-slave-3qdh krb5kdc[10533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:08 dist-test-slave-3qdh krb5kdc[10533](info): set up 2 sockets
Jun 19 14:57:08 dist-test-slave-3qdh krb5kdc[10533](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:57:09.743939 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:09 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:09.849255 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:32873
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:09.943681 10546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:09.943903 10546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:09.943964 10546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:09.947469 10546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:09.947538 10546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:09.947572 10546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:09.947619 10546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:09.947659 10546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:09.952399 10546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:09.953255 10546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:09.954064 10546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:09.961723 10552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:09.961809 10551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:09.961786 10546 server_base.cc:1034] running on GCE node
W20260619 14:57:09.961728 10554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:09.962549 10546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:09.963470 10546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:09.964635 10546 hybrid_clock.cc:648] HybridClock initialized: now 1781881029964616 us; error 38 us; skew 500 ppm
Jun 19 14:57:09 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881029, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:09.966894 10546 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:09.967693 10546 webserver.cc:460] Webserver started at http://127.10.71.190:35151/ using document root <none> and password file <none>
I20260619 14:57:09.968214 10546 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:09.968320 10546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:09.968534 10546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:09.970326 10546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "2a169ca6dc3b45f3aeede924e558c2ff"
format_stamp: "Formatted at 2026-06-19 14:57:09 on dist-test-slave-3qdh"
server_key: "3895a621afa5ef271aae70ea56eeff1e"
server_key_iv: "e226421c9a502f4944a2ffb7298a34d6"
server_key_version: "encryptionkey@0"
I20260619 14:57:09.970847 10546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "2a169ca6dc3b45f3aeede924e558c2ff"
format_stamp: "Formatted at 2026-06-19 14:57:09 on dist-test-slave-3qdh"
server_key: "3895a621afa5ef271aae70ea56eeff1e"
server_key_iv: "e226421c9a502f4944a2ffb7298a34d6"
server_key_version: "encryptionkey@0"
I20260619 14:57:09.974138 10546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:57:09.976133 10561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:09.977180 10546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:09.977301 10546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "2a169ca6dc3b45f3aeede924e558c2ff"
format_stamp: "Formatted at 2026-06-19 14:57:09 on dist-test-slave-3qdh"
server_key: "3895a621afa5ef271aae70ea56eeff1e"
server_key_iv: "e226421c9a502f4944a2ffb7298a34d6"
server_key_version: "encryptionkey@0"
I20260619 14:57:09.977397 10546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:09.985849 10546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:09.988090 10546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:09.988287 10546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:09.994513 10546 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:32873
I20260619 14:57:09.994524 10613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:32873 every 8 connection(s)
I20260619 14:57:09.995262 10546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:09.997515 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:10.001924 10614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff: Bootstrap starting.
I20260619 14:57:10.004016 10614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:10.004086 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 10546
I20260619 14:57:10.004228 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:10.004829 10614 log.cc:824] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:10.005589 10526 external_mini_cluster.cc:1428] Setting key 12bf8c0b858fc50d30845ac07cc4d534
I20260619 14:57:10.006593 10614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff: No bootstrap required, opened a new log
I20260619 14:57:10.010321 10614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 32873 } }
I20260619 14:57:10.010495 10614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:10.010550 10614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a169ca6dc3b45f3aeede924e558c2ff, State: Initialized, Role: FOLLOWER
I20260619 14:57:10.010926 10614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 32873 } }
I20260619 14:57:10.011051 10614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:10.011132 10614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:10.011222 10614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:10.011965 10614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 32873 } }
I20260619 14:57:10.012293 10614 leader_election.cc:302] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a169ca6dc3b45f3aeede924e558c2ff; no voters:
I20260619 14:57:10.012568 10614 leader_election.cc:288] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:10.012667 10619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:10.012856 10619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [term 1 LEADER]: Becoming Leader. State: Replica: 2a169ca6dc3b45f3aeede924e558c2ff, State: Running, Role: LEADER
I20260619 14:57:10.013105 10619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 32873 } }
I20260619 14:57:10.013594 10614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:10.014297 10620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 32873 } } }
I20260619 14:57:10.014420 10620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:10.014773 10626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:10.014742 10621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a169ca6dc3b45f3aeede924e558c2ff. Latest consensus state: current_term: 1 leader_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a169ca6dc3b45f3aeede924e558c2ff" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 32873 } } }
I20260619 14:57:10.014816 10621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:10.017516 10626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:10.022224 10626 catalog_manager.cc:1347] Generated new cluster ID: 6b3191d69461465385aa586c0a33aaf6
I20260619 14:57:10.022296 10626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:10.030313 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.008425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.10.71.190:32873)
0619 14:57:10.008840 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:10.008858 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:10.009888 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 14:57:10.012692 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:10.012708 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:10.013008 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:10.015785 (+ 2777us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:10.016323 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.017891 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.019604 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.020200 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.020388 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.024741 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:10.024747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:10.024761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:10.024801 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:10.027059 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.027544 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.027556 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.027562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.027688 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.028059 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.028063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.028065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.028592 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:10.028809 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:10.029334 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 14:57:10.029589 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260619 14:57:10.036864 10626 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:10.037631 10626 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:10.044816 10626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a169ca6dc3b45f3aeede924e558c2ff: Generated new TSK 0
I20260619 14:57:10.045289 10626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:10.071422 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:32873
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:10.169063 10640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:10.169299 10640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:10.169365 10640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:10.173066 10640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:10.173156 10640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:10.173239 10640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:10.177783 10640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:32873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:10.178562 10640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:10.179364 10640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:10.187691 10645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:10.187706 10648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:10.187697 10646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:10.187970 10640 server_base.cc:1034] running on GCE node
I20260619 14:57:10.188359 10640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:10.188894 10640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:10.190040 10640 hybrid_clock.cc:648] HybridClock initialized: now 1781881030190039 us; error 36 us; skew 500 ppm
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:10.192744 10640 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:10.193461 10640 webserver.cc:460] Webserver started at http://127.10.71.129:34325/ using document root <none> and password file <none>
I20260619 14:57:10.193907 10640 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:10.193954 10640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:10.194147 10640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:10.195748 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "7082cad4f1c04381a93b7cdd93c3f0e9"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "d33e7919413c1b7feef6aeab7a275d37"
server_key_iv: "9903d771f308d57a963a4e28c03ffcae"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.196269 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "7082cad4f1c04381a93b7cdd93c3f0e9"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "d33e7919413c1b7feef6aeab7a275d37"
server_key_iv: "9903d771f308d57a963a4e28c03ffcae"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.199329 10640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:57:10.201313 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.202343 10640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:10.202442 10640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "7082cad4f1c04381a93b7cdd93c3f0e9"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "d33e7919413c1b7feef6aeab7a275d37"
server_key_iv: "9903d771f308d57a963a4e28c03ffcae"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.202526 10640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:10.224741 10640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:10.227458 10640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:10.227639 10640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:10.228152 10640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:10.228878 10640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:10.228921 10640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.228955 10640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:10.229013 10640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.236830 10640 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:42163
I20260619 14:57:10.236857 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:42163 every 8 connection(s)
I20260619 14:57:10.237604 10640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:10.246286 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 10640
I20260619 14:57:10.246414 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:10.246730 10526 external_mini_cluster.cc:1428] Setting key f91453336b163155c4dc8481500d771d
I20260619 14:57:10.249322 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.239069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:34195 (local address 127.10.71.190:32873)
0619 14:57:10.239182 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:10.239185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:10.240221 (+ 1036us) server_negotiation.cc:407] Connection header received
0619 14:57:10.240984 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:10.240986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:10.241026 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:10.241118 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:10.242083 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.242599 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.243603 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.243805 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.243873 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.245886 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:10.245889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:10.245891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:10.245911 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:10.247380 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.247843 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.247847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.247848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.247929 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.248277 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.248279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.248280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.248464 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:10.248558 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:10.249094 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 14:57:10.249191 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:57:10.250324 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.239369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:32873 (local address 127.10.71.129:34195)
0619 14:57:10.240090 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:10.240125 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:10.240820 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:10.241231 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:10.241236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:10.241545 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:10.241965 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:10.241976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.242716 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.242719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:10.243499 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:10.243507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.243909 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.243911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:10.244030 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.244491 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:10.244504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:10.245730 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:10.247491 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:10.247495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:10.247505 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:10.247720 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:10.248024 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:10.248027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:10.248029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:10.248192 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:10.248558 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:10.248565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:10.248828 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 14:57:10.249049 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 14:57:10.249351 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":467,"threads_started":1}
I20260619 14:57:10.251380 10769 heartbeater.cc:347] Connected to a master server at 127.10.71.190:32873
I20260619 14:57:10.251571 10769 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:10.251981 10769 heartbeater.cc:511] Master 127.10.71.190:32873 requested a full tablet report, sending...
I20260619 14:57:10.253298 10578 ts_manager.cc:194] Registered new tserver with Master: 7082cad4f1c04381a93b7cdd93c3f0e9 (127.10.71.129:42163)
I20260619 14:57:10.254321 10578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:34195
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:10.269186 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:32873
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:10.366873 10774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:10.367056 10774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:10.367096 10774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:10.370903 10774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:10.370949 10774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:10.371043 10774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:10.375968 10774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:32873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:10.376844 10774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:10.377691 10774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:10.386061 10780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:10.386087 10779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:10.386063 10782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:10.386204 10774 server_base.cc:1034] running on GCE node
I20260619 14:57:10.386693 10774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:10.387218 10774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:10.388468 10774 hybrid_clock.cc:648] HybridClock initialized: now 1781881030388440 us; error 30 us; skew 500 ppm
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:10.391144 10774 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:10.391965 10774 webserver.cc:460] Webserver started at http://127.10.71.130:36037/ using document root <none> and password file <none>
I20260619 14:57:10.392541 10774 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:10.392645 10774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:10.392853 10774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:10.394550 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "872cad66590d4da487a335580b33bf72"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "bc302ade2c3eca614d04a0158b7c0a45"
server_key_iv: "70a21d5fc7328fb49de9523518afd216"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.395033 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "872cad66590d4da487a335580b33bf72"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "bc302ade2c3eca614d04a0158b7c0a45"
server_key_iv: "70a21d5fc7328fb49de9523518afd216"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.398063 10774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:10.399986 10789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.401039 10774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:10.401160 10774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "872cad66590d4da487a335580b33bf72"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "bc302ade2c3eca614d04a0158b7c0a45"
server_key_iv: "70a21d5fc7328fb49de9523518afd216"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.401257 10774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:10.410148 10774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:10.412915 10774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:10.413089 10774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:10.413591 10774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:10.414283 10774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:10.414340 10774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.414397 10774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:10.414429 10774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.422361 10774 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:45149
I20260619 14:57:10.422381 10902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:45149 every 8 connection(s)
I20260619 14:57:10.423013 10774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:10.423517 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 10774
I20260619 14:57:10.423626 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:10.423846 10526 external_mini_cluster.cc:1428] Setting key 961a00f40614e04b672e8a3fa156206f
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 14:57:10.436658 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.424567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56175 (local address 127.10.71.190:32873)
0619 14:57:10.424711 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:10.424715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:10.425953 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 14:57:10.427043 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:10.427046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:10.427124 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:10.427213 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:10.428492 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.429067 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.430761 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.430955 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.431026 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.433472 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:10.433476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:10.433477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:10.433497 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:10.434760 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.435233 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.435239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.435242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.435314 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.435608 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.435612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.435614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.435765 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:10.435867 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:10.436268 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 14:57:10.436420 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:57:10.437208 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.424894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:32873 (local address 127.10.71.130:56175)
0619 14:57:10.425795 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:10.425841 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:10.426868 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:10.427373 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:10.427393 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:10.427762 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:10.428357 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:10.428377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.429191 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.429194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:10.430646 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:10.430660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.431097 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.431102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:10.431304 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.431804 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:10.431826 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:10.433302 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:10.434887 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:10.434892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:10.434903 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:10.435144 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:10.435395 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:10.435400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:10.435402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:10.435520 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:10.435874 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:10.435879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:10.436048 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 14:57:10.436196 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:57:10.436448 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":520,"threads_started":1}
I20260619 14:57:10.438293 10903 heartbeater.cc:347] Connected to a master server at 127.10.71.190:32873
I20260619 14:57:10.438486 10903 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:10.438932 10903 heartbeater.cc:511] Master 127.10.71.190:32873 requested a full tablet report, sending...
I20260619 14:57:10.439888 10578 ts_manager.cc:194] Registered new tserver with Master: 872cad66590d4da487a335580b33bf72 (127.10.71.130:45149)
I20260619 14:57:10.440424 10578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:56175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:10.449079 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:32873
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:10.542243 10908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:10.542482 10908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:10.542546 10908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:10.546270 10908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:10.546329 10908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:10.546396 10908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:10.550974 10908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:32873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:10.551786 10908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:10.552666 10908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:10.560359 10916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:10.560403 10914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:10.560490 10913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:10.560648 10908 server_base.cc:1034] running on GCE node
I20260619 14:57:10.560979 10908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:10.561514 10908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:10.562659 10908 hybrid_clock.cc:648] HybridClock initialized: now 1781881030562641 us; error 34 us; skew 500 ppm
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:10.564980 10908 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:10.565773 10908 webserver.cc:460] Webserver started at http://127.10.71.131:36889/ using document root <none> and password file <none>
I20260619 14:57:10.566272 10908 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:10.566345 10908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:10.566530 10908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:10.568182 10908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "221dec6e880848f39db58864b3fe8dd5"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "ebaf7327a3450b550966a93143d36630"
server_key_iv: "e3d5b26ba8be7b8358165e185a3e4260"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.568756 10908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "221dec6e880848f39db58864b3fe8dd5"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "ebaf7327a3450b550966a93143d36630"
server_key_iv: "e3d5b26ba8be7b8358165e185a3e4260"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.571866 10908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:57:10.573858 10923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.574819 10908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:57:10.574986 10908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "221dec6e880848f39db58864b3fe8dd5"
format_stamp: "Formatted at 2026-06-19 14:57:10 on dist-test-slave-3qdh"
server_key: "ebaf7327a3450b550966a93143d36630"
server_key_iv: "e3d5b26ba8be7b8358165e185a3e4260"
server_key_version: "encryptionkey@0"
I20260619 14:57:10.575086 10908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:10.602988 10908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:10.605376 10908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:10.605522 10908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:10.605983 10908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:10.606678 10908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:10.606731 10908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.606786 10908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:10.606822 10908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:10.614971 10908 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:45119
I20260619 14:57:10.615005 11036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:45119 every 8 connection(s)
I20260619 14:57:10.615679 10908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:10.623813 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 10908
I20260619 14:57:10.623909 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:10.624131 10526 external_mini_cluster.cc:1428] Setting key c185590d896f217f234c831b69f94c1a
I20260619 14:57:10.627256 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.617139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:48843 (local address 127.10.71.190:32873)
0619 14:57:10.617263 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:10.617267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:10.618132 (+ 865us) server_negotiation.cc:407] Connection header received
0619 14:57:10.618977 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:10.618979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:10.619023 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:10.619102 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:10.620033 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.620575 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.621583 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.621792 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.621865 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.623989 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:10.623992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:10.623998 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:10.624017 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:10.625540 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.626019 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.626023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.626026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.626125 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.626478 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.626484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.626486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.626653 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:10.626724 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:10.627077 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 14:57:10.627154 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:57:10.627895 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.617453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:32873 (local address 127.10.71.131:48843)
0619 14:57:10.618005 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:10.618039 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:10.618810 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:10.619227 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:10.619233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:10.619519 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:10.619914 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:10.619925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.620662 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.620664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:10.621487 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:10.621495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.621901 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.621904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:10.622035 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.622478 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:10.622493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:10.623815 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:10.625619 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:10.625623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:10.625634 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:10.625927 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:10.626227 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:10.626230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:10.626232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:10.626387 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:10.626730 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:10.626734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:10.626900 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 14:57:10.627027 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 14:57:10.627230 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":99,"threads_started":1}
I20260619 14:57:10.628870 11037 heartbeater.cc:347] Connected to a master server at 127.10.71.190:32873
I20260619 14:57:10.629076 11037 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:10.629451 11037 heartbeater.cc:511] Master 127.10.71.190:32873 requested a full tablet report, sending...
I20260619 14:57:10.630239 10578 ts_manager.cc:194] Registered new tserver with Master: 221dec6e880848f39db58864b3fe8dd5 (127.10.71.131:45119)
I20260619 14:57:10.630693 10578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:48843
I20260619 14:57:10.637887 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:10.660408 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.652446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.10.71.129:42163)
0619 14:57:10.652715 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:10.652726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:10.652750 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:57:10.652802 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:10.652807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:10.652950 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:10.653110 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:10.653697 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.654418 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.655236 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.655574 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.655673 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.657618 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:10.657625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:10.657633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:10.657654 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:10.658859 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.659233 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.659237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.659238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.659304 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.659533 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.659537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.659539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.659736 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:10.659926 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:10.660102 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:57:10.660231 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
W20260619 14:57:10.661155 10683 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43406
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[10533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:10.683389 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:10.675890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.10.71.129:42163)
0619 14:57:10.676009 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:10.676013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:10.676141 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:57:10.676285 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:10.676288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:10.676358 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:10.676436 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:10.677055 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.677510 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.678343 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:10.678557 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:10.678661 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:10.680925 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:10.680929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:10.680932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:10.680957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:10.682197 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.682549 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.682553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.682554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.682639 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:10.682833 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:10.682835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:10.682836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:10.682973 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:10.683081 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:10.683208 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:57:10.683285 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:10.684114 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 10640
I20260619 14:57:10.689152 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 10774
I20260619 14:57:10.693794 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 10908
I20260619 14:57:10.698110 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 10546
2026-06-19T14:57:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1938 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[11051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[11051](info): set up 2 sockets
Jun 19 14:57:10 dist-test-slave-3qdh krb5kdc[11051](info): commencing operation
krb5kdc: starting...
W20260619 14:57:11.663719 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:11 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:11.764743 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39345
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:39345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:11.859648 11064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:11.859886 11064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:11.859947 11064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:11.863667 11064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:11.863708 11064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:11.863724 11064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:11.863762 11064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:11.863796 11064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:11.868398 11064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:39345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:11.869184 11064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:11.869956 11064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:11.877177 11072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:11.877185 11069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:11.877233 11064 server_base.cc:1034] running on GCE node
W20260619 14:57:11.877185 11070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:11.877799 11064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:11.878697 11064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:11.879839 11064 hybrid_clock.cc:648] HybridClock initialized: now 1781881031879824 us; error 25 us; skew 500 ppm
Jun 19 14:57:11 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881031, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:11.882198 11064 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:11.882893 11064 webserver.cc:460] Webserver started at http://127.10.71.190:46185/ using document root <none> and password file <none>
I20260619 14:57:11.883337 11064 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:11.883410 11064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:11.883595 11064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:11.885224 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "a110371ab2564dd597946785203bcafa"
format_stamp: "Formatted at 2026-06-19 14:57:11 on dist-test-slave-3qdh"
server_key: "c0884b2a9f23597c93a9ad79c95d3f95"
server_key_iv: "7514f7b0a92eb61e6b1375b614843fc8"
server_key_version: "encryptionkey@0"
I20260619 14:57:11.885704 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "a110371ab2564dd597946785203bcafa"
format_stamp: "Formatted at 2026-06-19 14:57:11 on dist-test-slave-3qdh"
server_key: "c0884b2a9f23597c93a9ad79c95d3f95"
server_key_iv: "7514f7b0a92eb61e6b1375b614843fc8"
server_key_version: "encryptionkey@0"
I20260619 14:57:11.888746 11064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:11.890715 11079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:11.891743 11064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:11.891865 11064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "a110371ab2564dd597946785203bcafa"
format_stamp: "Formatted at 2026-06-19 14:57:11 on dist-test-slave-3qdh"
server_key: "c0884b2a9f23597c93a9ad79c95d3f95"
server_key_iv: "7514f7b0a92eb61e6b1375b614843fc8"
server_key_version: "encryptionkey@0"
I20260619 14:57:11.891959 11064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:11.907881 11064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:11.910326 11064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:11.910496 11064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:11.916381 11064 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:39345
I20260619 14:57:11.916431 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:39345 every 8 connection(s)
I20260619 14:57:11.917138 11064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:11.918823 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11064
I20260619 14:57:11.918943 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:11.919163 10526 external_mini_cluster.cc:1428] Setting key eaa26100b5097356b9838753e37715bf
I20260619 14:57:11.919489 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:11.925647 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa: Bootstrap starting.
Jun 19 14:57:11 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:11.927790 11132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:11.928580 11132 log.cc:824] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:11.930328 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa: No bootstrap required, opened a new log
I20260619 14:57:11.931771 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:11.920279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.10.71.190:39345)
0619 14:57:11.920678 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:11.920696 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:11.920733 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:57:11.921492 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:11.921511 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:11.921883 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:11.922218 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:11.922787 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:11.924101 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:11.924879 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:11.925455 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:11.925588 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:11.927827 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:11.927831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:11.927840 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:11.927880 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:11.929309 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:11.929743 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:11.929746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:11.929750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:11.929846 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:11.930082 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:11.930085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:11.930087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:11.930380 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:11.930555 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:11.930779 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:57:11.930999 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":101,"threads_started":1}
I20260619 14:57:11.933275 11132 raft_consensus.cc:348] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110371ab2564dd597946785203bcafa" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39345 } }
I20260619 14:57:11.933504 11132 raft_consensus.cc:374] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:11.933560 11132 raft_consensus.cc:729] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110371ab2564dd597946785203bcafa, State: Initialized, Role: FOLLOWER
I20260619 14:57:11.933977 11132 consensus_queue.cc:260] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110371ab2564dd597946785203bcafa" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39345 } }
I20260619 14:57:11.934113 11132 raft_consensus.cc:388] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:11.934211 11132 raft_consensus.cc:482] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:11.934305 11132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:11.935089 11132 raft_consensus.cc:504] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110371ab2564dd597946785203bcafa" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39345 } }
I20260619 14:57:11.935384 11132 leader_election.cc:302] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a110371ab2564dd597946785203bcafa; no voters:
I20260619 14:57:11.935676 11132 leader_election.cc:288] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:11.935747 11137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:11.935921 11137 raft_consensus.cc:686] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [term 1 LEADER]: Becoming Leader. State: Replica: a110371ab2564dd597946785203bcafa, State: Running, Role: LEADER
I20260619 14:57:11.936234 11137 consensus_queue.cc:237] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110371ab2564dd597946785203bcafa" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39345 } }
I20260619 14:57:11.936759 11132 sys_catalog.cc:564] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:11.937757 11139 sys_catalog.cc:455] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a110371ab2564dd597946785203bcafa. Latest consensus state: current_term: 1 leader_uuid: "a110371ab2564dd597946785203bcafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110371ab2564dd597946785203bcafa" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39345 } } }
I20260619 14:57:11.937762 11138 sys_catalog.cc:455] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a110371ab2564dd597946785203bcafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110371ab2564dd597946785203bcafa" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39345 } } }
I20260619 14:57:11.937983 11139 sys_catalog.cc:458] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:11.937999 11138 sys_catalog.cc:458] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:11.938891 11147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:11.941104 11147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:11.946137 11147 catalog_manager.cc:1347] Generated new cluster ID: 8d7e896b083e44128d76c9ea2edf60c9
I20260619 14:57:11.946198 11147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:11.980540 11147 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:11.981376 11147 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:11.995837 11147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a110371ab2564dd597946785203bcafa: Generated new TSK 0
I20260619 14:57:11.996562 11147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:12.025302 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39345
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:12.132570 11158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:12.132793 11158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:12.132877 11158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:12.137224 11158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:12.137279 11158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:12.137388 11158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:12.142524 11158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:12.143396 11158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:12.144338 11158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:12.153120 11163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:12.153120 11166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:12.153304 11158 server_base.cc:1034] running on GCE node
W20260619 14:57:12.153101 11164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:12.153864 11158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:12.154448 11158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:12.155634 11158 hybrid_clock.cc:648] HybridClock initialized: now 1781881032155620 us; error 32 us; skew 500 ppm
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881032, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:12.158275 11158 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:12.159101 11158 webserver.cc:460] Webserver started at http://127.10.71.129:36307/ using document root <none> and password file <none>
I20260619 14:57:12.159605 11158 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:12.159695 11158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:12.159920 11158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:12.161820 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "d278bc9093564f288d53c9268e2d95fc"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "20f02f9961926bfeb783fa9c008184aa"
server_key_iv: "40e858f88f1feae477725304d0523ab6"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.162380 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "d278bc9093564f288d53c9268e2d95fc"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "20f02f9961926bfeb783fa9c008184aa"
server_key_iv: "40e858f88f1feae477725304d0523ab6"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.165855 11158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:57:12.168099 11173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.169216 11158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:12.169356 11158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "d278bc9093564f288d53c9268e2d95fc"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "20f02f9961926bfeb783fa9c008184aa"
server_key_iv: "40e858f88f1feae477725304d0523ab6"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.169493 11158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:12.192142 11158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:12.194912 11158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:12.195178 11158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:12.195659 11158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:12.196427 11158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:12.196486 11158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.196522 11158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:12.196570 11158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.204797 11158 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:44247
I20260619 14:57:12.204813 11286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:44247 every 8 connection(s)
I20260619 14:57:12.205492 11158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:12.210556 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11158
I20260619 14:57:12.210686 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:12.210914 10526 external_mini_cluster.cc:1428] Setting key 0ada05b34bb841d49da9d0b62aabae80
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881032, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:12.217442 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.206979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:39345 (local address 127.10.71.190:39345)
0619 14:57:12.207129 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.207134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.207843 (+ 709us) server_negotiation.cc:407] Connection header received
0619 14:57:12.208649 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.208652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.208699 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.208772 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:12.209689 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.210158 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.211297 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.211612 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.211717 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.213974 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:12.213981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:12.213985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:12.214014 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:12.215446 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.216003 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.216006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.216007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.216075 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.216448 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.216450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.216451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.216603 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:12.216670 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.217240 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.217337 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:12.218403 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.207325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39345 (local address 127.10.71.129:39345)
0619 14:57:12.207712 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.207745 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.208469 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.208905 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.208911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.209190 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:12.209559 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.209584 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.210258 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.210260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.211155 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.211164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.211771 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.211775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.211940 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.212605 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:12.212618 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:12.213826 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:12.215570 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:12.215577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:12.215590 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:12.215898 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:12.216176 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:12.216180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:12.216181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:12.216360 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:12.216705 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:12.216714 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:12.216978 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.217194 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.217497 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260619 14:57:12.219440 11287 heartbeater.cc:347] Connected to a master server at 127.10.71.190:39345
I20260619 14:57:12.219655 11287 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:12.220132 11287 heartbeater.cc:511] Master 127.10.71.190:39345 requested a full tablet report, sending...
I20260619 14:57:12.221632 11096 ts_manager.cc:194] Registered new tserver with Master: d278bc9093564f288d53c9268e2d95fc (127.10.71.129:44247)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:12.222978 11096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:39345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:12.236732 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39345
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:12.333254 11292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:12.333477 11292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:12.333542 11292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:12.337296 11292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:12.337355 11292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:12.337445 11292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:12.342301 11292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:12.343152 11292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:12.343966 11292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:12.352532 11300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:12.352602 11298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:12.352610 11297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:12.352660 11292 server_base.cc:1034] running on GCE node
I20260619 14:57:12.353165 11292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:12.353677 11292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:12.354846 11292 hybrid_clock.cc:648] HybridClock initialized: now 1781881032354841 us; error 26 us; skew 500 ppm
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881032, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:12.357246 11292 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:12.357930 11292 webserver.cc:460] Webserver started at http://127.10.71.130:35515/ using document root <none> and password file <none>
I20260619 14:57:12.358403 11292 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:12.358484 11292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:12.358682 11292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:12.360459 11292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "e9c571696d334ce78c4f9c7d1543386a"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "0a454cf967144bef42b66a3fe2b42c82"
server_key_iv: "cb0192cfb980980fa3b153e7543286ad"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.360963 11292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "e9c571696d334ce78c4f9c7d1543386a"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "0a454cf967144bef42b66a3fe2b42c82"
server_key_iv: "cb0192cfb980980fa3b153e7543286ad"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.364126 11292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:57:12.366262 11307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.367269 11292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:57:12.367393 11292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "e9c571696d334ce78c4f9c7d1543386a"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "0a454cf967144bef42b66a3fe2b42c82"
server_key_iv: "cb0192cfb980980fa3b153e7543286ad"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.367486 11292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:12.381171 11292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:12.383522 11292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:12.383697 11292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:12.384167 11292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:12.384868 11292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:12.384938 11292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.384997 11292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:12.385043 11292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.393066 11292 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:46263
I20260619 14:57:12.393102 11420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:46263 every 8 connection(s)
I20260619 14:57:12.393791 11292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881032, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:12.401587 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11292
I20260619 14:57:12.401701 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:12.401942 10526 external_mini_cluster.cc:1428] Setting key 206f66d34d3e61c5689c4015c89e06a8
I20260619 14:57:12.405633 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.395356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:39477 (local address 127.10.71.190:39345)
0619 14:57:12.395482 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.395486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.396492 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 14:57:12.397178 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.397180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.397222 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.397287 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:12.398202 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.398640 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.399516 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.399707 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.399772 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.402225 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:12.402229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:12.402232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:12.402255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:12.403689 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.404143 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.404149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.404154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.404243 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.404584 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.404589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.404593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.404817 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:12.404918 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.405442 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.405522 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:57:12.406459 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.395715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39345 (local address 127.10.71.130:39477)
0619 14:57:12.396350 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.396398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.397022 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.397416 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.397422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.397704 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:12.398078 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.398090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.398727 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.398729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.399425 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.399432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.399810 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.399812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.399925 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.400443 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:12.400456 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:12.402067 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:12.403799 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:12.403803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:12.403811 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:12.404049 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:12.404387 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:12.404389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:12.404390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:12.404500 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:12.404928 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:12.404935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:12.405223 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.405405 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.405679 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":113,"threads_started":1}
I20260619 14:57:12.407456 11421 heartbeater.cc:347] Connected to a master server at 127.10.71.190:39345
I20260619 14:57:12.407663 11421 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:12.408206 11421 heartbeater.cc:511] Master 127.10.71.190:39345 requested a full tablet report, sending...
I20260619 14:57:12.409276 11096 ts_manager.cc:194] Registered new tserver with Master: e9c571696d334ce78c4f9c7d1543386a (127.10.71.130:46263)
I20260619 14:57:12.409777 11096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:39477
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:12.426658 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39345
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:12.526401 11426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:12.526615 11426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:12.526690 11426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:12.530431 11426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:12.530490 11426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:12.530611 11426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:12.535291 11426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:12.536147 11426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:12.537056 11426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:12.546013 11432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:12.545984 11431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:12.545985 11434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:12.546185 11426 server_base.cc:1034] running on GCE node
I20260619 14:57:12.546661 11426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:12.547214 11426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:12.548378 11426 hybrid_clock.cc:648] HybridClock initialized: now 1781881032548354 us; error 26 us; skew 500 ppm
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881032, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:12.551031 11426 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:12.551889 11426 webserver.cc:460] Webserver started at http://127.10.71.131:46411/ using document root <none> and password file <none>
I20260619 14:57:12.552501 11426 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:12.552616 11426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:12.552824 11426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:12.554579 11426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "22ea1678d2dd45387b5bc65eadacfae9"
server_key_iv: "a53767be53b1015a2fc17682ba61236e"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.555115 11426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "22ea1678d2dd45387b5bc65eadacfae9"
server_key_iv: "a53767be53b1015a2fc17682ba61236e"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.558462 11426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 14:57:12.560570 11441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.561646 11426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:12.561780 11426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20"
format_stamp: "Formatted at 2026-06-19 14:57:12 on dist-test-slave-3qdh"
server_key: "22ea1678d2dd45387b5bc65eadacfae9"
server_key_iv: "a53767be53b1015a2fc17682ba61236e"
server_key_version: "encryptionkey@0"
I20260619 14:57:12.561885 11426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:12.579438 11426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:12.582070 11426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:12.582271 11426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:12.582823 11426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:12.583575 11426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:12.583643 11426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.583705 11426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:12.583768 11426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:12.592669 11426 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:36527
I20260619 14:57:12.592721 11554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:36527 every 8 connection(s)
I20260619 14:57:12.593529 11426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881032, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:12.601475 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11426
I20260619 14:57:12.601574 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:12.601859 10526 external_mini_cluster.cc:1428] Setting key 08c03c52f8f76f125171ec748786d0c3
I20260619 14:57:12.605576 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.595209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:39865 (local address 127.10.71.190:39345)
0619 14:57:12.595338 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.595343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.596156 (+ 813us) server_negotiation.cc:407] Connection header received
0619 14:57:12.596932 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.596934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.596977 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.597043 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:12.597992 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.598474 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.599493 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.599702 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.599778 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.602133 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:12.602137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:12.602138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:12.602159 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:12.603759 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.604243 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.604247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.604249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.604354 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.604664 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.604667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.604668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.604850 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:12.604939 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.605334 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.605434 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:12.606217 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.595560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39345 (local address 127.10.71.131:39865)
0619 14:57:12.596027 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.596061 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.596742 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.597180 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.597185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.597468 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:12.597885 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.597897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.598568 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.598570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.599396 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.599404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.599815 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.599818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.599935 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.600460 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:12.600474 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:12.601956 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:12.603915 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:12.603920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:12.603937 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:12.604147 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:12.604445 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:12.604447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:12.604449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:12.604567 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:12.604948 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:12.604955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:12.605146 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.605297 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.605540 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260619 14:57:12.607136 11555 heartbeater.cc:347] Connected to a master server at 127.10.71.190:39345
I20260619 14:57:12.607319 11555 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:12.607661 11555 heartbeater.cc:511] Master 127.10.71.190:39345 requested a full tablet report, sending...
I20260619 14:57:12.608538 11096 ts_manager.cc:194] Registered new tserver with Master: 7f5bbb943f1f4f6db46e3f192a1a1f20 (127.10.71.131:36527)
I20260619 14:57:12.609023 11096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:39865
I20260619 14:57:12.615240 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:57:12.624214 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.618057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.10.71.190:39345)
0619 14:57:12.618156 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.618159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.618272 (+ 113us) server_negotiation.cc:407] Connection header received
0619 14:57:12.618312 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.618314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.618349 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.618409 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:12.619014 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.619531 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.620256 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.620523 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.620595 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.621682 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:12.621685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:12.621686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:12.621716 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:12.622998 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.623337 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.623339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.623340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.623405 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:12.623611 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:12.623613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:12.623614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:12.623785 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:12.623878 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.624023 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.624116 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:57:12.628409 11096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:12.631116 11096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:12.645102 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.639354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36527 (local address 127.0.0.1:43166)
0619 14:57:12.640334 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.640348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.640432 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.640875 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.640878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.640894 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:12.640987 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.640996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.641950 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.641956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.643456 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.643466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.644742 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.644745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.644827 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.644845 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.644908 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.644964 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":48,"thread_start_us":88,"threads_started":1}
I20260619 14:57:12.645105 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.639081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.10.71.129:44247)
0619 14:57:12.639668 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.639679 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.639854 (+ 175us) server_negotiation.cc:407] Connection header received
0619 14:57:12.640164 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.640170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.640382 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.640513 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:12.640961 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.642141 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.643754 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.644290 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.644465 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.644619 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.644862 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.644967 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":118,"threads_started":1}
I20260619 14:57:12.645105 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.639190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:46263 (local address 127.0.0.1:48116)
0619 14:57:12.639937 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.639950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.640056 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.640553 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.640561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.640582 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:12.640712 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.640718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.642282 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.642285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.643487 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.643496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.644571 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.644574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.644660 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.644708 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.644788 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.644920 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":77,"thread_start_us":86,"threads_started":1}
I20260619 14:57:12.645443 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.639580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.10.71.130:46263)
0619 14:57:12.640080 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.640093 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.640115 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:57:12.640177 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.640181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.640386 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.640853 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:12.641018 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.642141 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.643643 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.644433 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.645020 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.645162 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.645216 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.645309 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":113,"threads_started":1}
I20260619 14:57:12.645473 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.638862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:44247 (local address 127.0.0.1:41384)
0619 14:57:12.639724 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.639772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.639954 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.640708 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.640711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.640726 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:12.640836 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.640842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.642461 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.642464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.643647 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.643654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.644463 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.644466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.644704 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.644731 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.644830 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.644938 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260619 14:57:12.645613 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.639562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.10.71.131:36527)
0619 14:57:12.639863 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.639879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.640527 (+ 648us) server_negotiation.cc:407] Connection header received
0619 14:57:12.640603 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.640616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.640749 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.640896 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:12.641120 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.641824 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.643593 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.644433 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.644547 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.644684 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.645381 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.645480 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":123,"threads_started":1}
I20260619 14:57:12.647831 11489 tablet_service.cc:1467] Processing CreateTablet for tablet cd08c3b944e6496ca68830dadc107495 (DEFAULT_TABLE table=test-table [id=5d41ef941d994198808322252cf32fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:12.647835 11355 tablet_service.cc:1467] Processing CreateTablet for tablet cd08c3b944e6496ca68830dadc107495 (DEFAULT_TABLE table=test-table [id=5d41ef941d994198808322252cf32fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:12.648680 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd08c3b944e6496ca68830dadc107495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:12.648687 11489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd08c3b944e6496ca68830dadc107495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:12.654126 11572 tablet_bootstrap.cc:492] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: Bootstrap starting.
I20260619 14:57:12.654655 11221 tablet_service.cc:1467] Processing CreateTablet for tablet cd08c3b944e6496ca68830dadc107495 (DEFAULT_TABLE table=test-table [id=5d41ef941d994198808322252cf32fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:12.655493 11221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd08c3b944e6496ca68830dadc107495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:12.656355 11572 tablet_bootstrap.cc:654] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:12.656625 11573 tablet_bootstrap.cc:492] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: Bootstrap starting.
I20260619 14:57:12.657181 11572 log.cc:824] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:12.658432 11573 tablet_bootstrap.cc:654] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:12.659078 11572 tablet_bootstrap.cc:492] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: No bootstrap required, opened a new log
I20260619 14:57:12.659242 11572 ts_tablet_manager.cc:1397] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:12.659507 11573 log.cc:824] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:12.661095 11576 tablet_bootstrap.cc:492] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: Bootstrap starting.
I20260619 14:57:12.661329 11573 tablet_bootstrap.cc:492] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: No bootstrap required, opened a new log
I20260619 14:57:12.661509 11573 ts_tablet_manager.cc:1397] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:12.663077 11572 raft_consensus.cc:348] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.663331 11572 raft_consensus.cc:374] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:12.663403 11572 raft_consensus.cc:729] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c571696d334ce78c4f9c7d1543386a, State: Initialized, Role: FOLLOWER
I20260619 14:57:12.663817 11576 tablet_bootstrap.cc:654] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:12.663806 11572 consensus_queue.cc:260] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.664429 11421 heartbeater.cc:503] Master 127.10.71.190:39345 was elected leader, sending a full tablet report...
I20260619 14:57:12.664618 11576 log.cc:824] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:12.664721 11572 ts_tablet_manager.cc:1428] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:57:12.665143 11573 raft_consensus.cc:348] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.665377 11573 raft_consensus.cc:374] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:12.665459 11573 raft_consensus.cc:729] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5bbb943f1f4f6db46e3f192a1a1f20, State: Initialized, Role: FOLLOWER
I20260619 14:57:12.665841 11573 consensus_queue.cc:260] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.666388 11555 heartbeater.cc:503] Master 127.10.71.190:39345 was elected leader, sending a full tablet report...
I20260619 14:57:12.666391 11576 tablet_bootstrap.cc:492] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: No bootstrap required, opened a new log
I20260619 14:57:12.666566 11576 ts_tablet_manager.cc:1397] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 14:57:12.666676 11573 ts_tablet_manager.cc:1428] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:57:12.670131 11576 raft_consensus.cc:348] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.670315 11576 raft_consensus.cc:374] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:12.670380 11576 raft_consensus.cc:729] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d278bc9093564f288d53c9268e2d95fc, State: Initialized, Role: FOLLOWER
I20260619 14:57:12.670727 11576 consensus_queue.cc:260] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.671250 11287 heartbeater.cc:503] Master 127.10.71.190:39345 was elected leader, sending a full tablet report...
I20260619 14:57:12.671519 11576 ts_tablet_manager.cc:1428] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 14:57:12.706523 11288 tablet.cc:2369] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:57:12.844452 11556 tablet.cc:2369] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:57:12.894877 11422 tablet.cc:2369] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:12.974905 11580 raft_consensus.cc:482] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:12.975116 11580 raft_consensus.cc:504] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.976163 11580 leader_election.cc:288] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c571696d334ce78c4f9c7d1543386a (127.10.71.130:46263), 7f5bbb943f1f4f6db46e3f192a1a1f20 (127.10.71.131:36527)
I20260619 14:57:12.979943 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.976570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:44589 (local address 127.10.71.131:36527)
0619 14:57:12.976719 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.976723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.976869 (+ 146us) server_negotiation.cc:407] Connection header received
0619 14:57:12.977027 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.977035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.977073 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.977151 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:12.977423 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.977952 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.978979 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.979377 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.979512 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.979565 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.979746 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.979811 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:57:12.979960 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.976433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36527 (local address 127.10.71.129:44589)
0619 14:57:12.976777 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.976789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.976930 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.977181 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.977184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.977198 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:12.977319 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.977324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.978063 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.978067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.978855 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.978863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.979509 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.979513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.979608 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.979637 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.979718 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.979789 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":72,"threads_started":1}
I20260619 14:57:12.979983 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.976505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:46595 (local address 127.10.71.130:46263)
0619 14:57:12.976673 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.976678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.976940 (+ 262us) server_negotiation.cc:407] Connection header received
0619 14:57:12.977098 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.977101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.977172 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.977265 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:12.977538 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.978026 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.979051 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.979432 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.979560 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.979607 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:12.979799 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 14:57:12.979863 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:57:12.980005 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.976433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:46263 (local address 127.10.71.129:46595)
0619 14:57:12.976846 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:12.976858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:12.976966 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:12.977273 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:12.977275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:12.977297 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:12.977409 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.977414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.978139 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.978144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.978927 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:12.978935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.979553 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.979557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:12.979668 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:12.979685 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:57:12.979782 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:57:12.979849 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":36,"thread_start_us":137,"threads_started":1}
I20260619 14:57:12.980661 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd08c3b944e6496ca68830dadc107495" candidate_uuid: "d278bc9093564f288d53c9268e2d95fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c571696d334ce78c4f9c7d1543386a" is_pre_election: true
I20260619 14:57:12.980684 11509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd08c3b944e6496ca68830dadc107495" candidate_uuid: "d278bc9093564f288d53c9268e2d95fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" is_pre_election: true
I20260619 14:57:12.980963 11375 raft_consensus.cc:2454] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d278bc9093564f288d53c9268e2d95fc in term 0.
I20260619 14:57:12.980975 11509 raft_consensus.cc:2454] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d278bc9093564f288d53c9268e2d95fc in term 0.
I20260619 14:57:12.981438 11176 leader_election.cc:302] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d278bc9093564f288d53c9268e2d95fc, e9c571696d334ce78c4f9c7d1543386a; no voters:
I20260619 14:57:12.981706 11580 raft_consensus.cc:2781] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:12.981793 11580 raft_consensus.cc:482] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:12.981835 11580 raft_consensus.cc:3037] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:12.982832 11580 raft_consensus.cc:504] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.983176 11580 leader_election.cc:288] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [CANDIDATE]: Term 1 election: Requested vote from peers e9c571696d334ce78c4f9c7d1543386a (127.10.71.130:46263), 7f5bbb943f1f4f6db46e3f192a1a1f20 (127.10.71.131:36527)
I20260619 14:57:12.983496 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd08c3b944e6496ca68830dadc107495" candidate_uuid: "d278bc9093564f288d53c9268e2d95fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c571696d334ce78c4f9c7d1543386a"
I20260619 14:57:12.983598 11509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd08c3b944e6496ca68830dadc107495" candidate_uuid: "d278bc9093564f288d53c9268e2d95fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20"
I20260619 14:57:12.983632 11375 raft_consensus.cc:3037] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:12.983738 11509 raft_consensus.cc:3037] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:12.984735 11375 raft_consensus.cc:2454] T cd08c3b944e6496ca68830dadc107495 P e9c571696d334ce78c4f9c7d1543386a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d278bc9093564f288d53c9268e2d95fc in term 1.
I20260619 14:57:12.984815 11509 raft_consensus.cc:2454] T cd08c3b944e6496ca68830dadc107495 P 7f5bbb943f1f4f6db46e3f192a1a1f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d278bc9093564f288d53c9268e2d95fc in term 1.
I20260619 14:57:12.985037 11176 leader_election.cc:302] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d278bc9093564f288d53c9268e2d95fc, e9c571696d334ce78c4f9c7d1543386a; no voters:
I20260619 14:57:12.985219 11580 raft_consensus.cc:2781] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:12.985430 11580 raft_consensus.cc:686] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [term 1 LEADER]: Becoming Leader. State: Replica: d278bc9093564f288d53c9268e2d95fc, State: Running, Role: LEADER
I20260619 14:57:12.985754 11580 consensus_queue.cc:237] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } }
I20260619 14:57:12.988878 11095 catalog_manager.cc:5495] T cd08c3b944e6496ca68830dadc107495 P d278bc9093564f288d53c9268e2d95fc reported cstate change: term changed from 0 to 1, leader changed from <none> to d278bc9093564f288d53c9268e2d95fc (127.10.71.129). New cstate: current_term: 1 leader_uuid: "d278bc9093564f288d53c9268e2d95fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d278bc9093564f288d53c9268e2d95fc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c571696d334ce78c4f9c7d1543386a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f5bbb943f1f4f6db46e3f192a1a1f20" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36527 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:57:12 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:13.005658 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:12.997701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.10.71.129:44247)
0619 14:57:12.997821 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:12.997826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:12.997962 (+ 136us) server_negotiation.cc:407] Connection header received
0619 14:57:12.998052 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:12.998054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:12.998091 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:12.998152 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:12.998797 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:12.999238 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:12.999977 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:13.000176 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:13.000258 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:13.002630 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:13.002638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:13.002649 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:13.002670 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:13.004024 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:13.004468 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:13.004473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:13.004474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:13.004544 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:13.004815 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:13.004820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:13.004822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:13.005041 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:13.005259 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:13.005402 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:57:13.005513 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881033, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881033, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:13.030303 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:13.022623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.10.71.129:44247)
0619 14:57:13.022731 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:13.022735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:13.022881 (+ 146us) server_negotiation.cc:407] Connection header received
0619 14:57:13.023017 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:13.023020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:13.023061 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:13.023147 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:13.023845 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:13.024375 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:13.025155 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:13.025371 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:13.025470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:13.027562 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:13.027571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:13.027579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:13.027597 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:13.028784 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:13.029202 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:13.029207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:13.029209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:13.029280 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:13.029533 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:13.029538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:13.029539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:13.029686 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:13.029836 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:13.030064 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:57:13.030159 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:13.053414 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:13.045591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.10.71.129:44247)
0619 14:57:13.045705 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:13.045709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:13.045865 (+ 156us) server_negotiation.cc:407] Connection header received
0619 14:57:13.045953 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:13.045955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:13.045986 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:13.046060 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:13.046664 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:13.047103 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:13.047867 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:13.048054 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:13.048129 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:13.050498 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:13.050501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:13.050503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:13.050525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:13.051957 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:13.052356 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:13.052360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:13.052362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:13.052458 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:13.052728 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:13.052731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:13.052732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:13.052885 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:13.053005 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:13.053224 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:57:13.053309 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:57:13.054653 11201 tablet_service.cc:2908] Rejecting scan request for tablet cd08c3b944e6496ca68830dadc107495: Uninitialized: clean time has not yet been initialized
I20260619 14:57:13.056437 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11158
I20260619 14:57:13.062331 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11292
I20260619 14:57:13.067610 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11426
I20260619 14:57:13.072636 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11064
2026-06-19T14:57:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2375 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11597](info): set up 2 sockets
Jun 19 14:57:13 dist-test-slave-3qdh krb5kdc[11597](info): commencing operation
krb5kdc: starting...
W20260619 14:57:14.067651 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:14.172119 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37899
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:14.269701 11610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:14.269903 11610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:14.269940 11610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:14.273574 11610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:14.273617 11610 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:57:14.273633 11610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:14.273651 11610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:14.273666 11610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:14.273682 11610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:14.278090 11610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:37899
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:14.278800 11610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:14.279546 11610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:14.286705 11615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.286856 11610 server_base.cc:1034] running on GCE node
W20260619 14:57:14.286717 11616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:14.286718 11618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.287478 11610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:14.288551 11610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:14.289743 11610 hybrid_clock.cc:648] HybridClock initialized: now 1781881034289716 us; error 42 us; skew 500 ppm
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:14.296720 11610 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:14.297518 11610 webserver.cc:460] Webserver started at http://127.10.71.190:43531/ using document root <none> and password file <none>
I20260619 14:57:14.298010 11610 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:14.298099 11610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:14.298292 11610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:14.299979 11610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "03aa90806d9b43ad847635f0360c0494"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "d75836e1349540ac5a093d7831082d74"
server_key_iv: "7b8ec08b5dd0b2d665ad8b949e23ccc3"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.300529 11610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "03aa90806d9b43ad847635f0360c0494"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "d75836e1349540ac5a093d7831082d74"
server_key_iv: "7b8ec08b5dd0b2d665ad8b949e23ccc3"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.303673 11610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:57:14.305742 11625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.306715 11610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:14.306855 11610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "03aa90806d9b43ad847635f0360c0494"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "d75836e1349540ac5a093d7831082d74"
server_key_iv: "7b8ec08b5dd0b2d665ad8b949e23ccc3"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.306960 11610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:14.320899 11610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:14.323344 11610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:14.323514 11610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:14.330708 11610 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:37899
I20260619 14:57:14.330721 11687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:37899 every 8 connection(s)
I20260619 14:57:14.331547 11610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:14.334014 11688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:14.336891 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11610
I20260619 14:57:14.336997 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:14.337248 10526 external_mini_cluster.cc:1428] Setting key fd721ccb1ebf6a86702317521b22075e
I20260619 14:57:14.339412 11688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: Bootstrap starting.
I20260619 14:57:14.341673 11688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:14.342480 11688 log.cc:824] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:14.344370 11688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: No bootstrap required, opened a new log
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:14.347766 11688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03aa90806d9b43ad847635f0360c0494" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37899 } }
I20260619 14:57:14.347958 11688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:14.348034 11688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03aa90806d9b43ad847635f0360c0494, State: Initialized, Role: FOLLOWER
I20260619 14:57:14.348521 11688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03aa90806d9b43ad847635f0360c0494" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37899 } }
I20260619 14:57:14.348676 11688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:14.348757 11688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:14.348842 11688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:14.349800 11688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03aa90806d9b43ad847635f0360c0494" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37899 } }
I20260619 14:57:14.350106 11688 leader_election.cc:302] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03aa90806d9b43ad847635f0360c0494; no voters:
I20260619 14:57:14.350375 11688 leader_election.cc:288] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:14.350485 11693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:14.350679 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.338442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.10.71.190:37899)
0619 14:57:14.338833 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.338853 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.338884 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:57:14.339448 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.339487 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.339787 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.340025 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.340893 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.341917 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.343015 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.343484 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.343601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.346151 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.346156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.346165 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.346191 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.347686 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.348210 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.348215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.348220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.348388 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.348650 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.348653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.348655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.349127 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.349305 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.349520 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.349813 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260619 14:57:14.350749 11693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [term 1 LEADER]: Becoming Leader. State: Replica: 03aa90806d9b43ad847635f0360c0494, State: Running, Role: LEADER
I20260619 14:57:14.351209 11693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03aa90806d9b43ad847635f0360c0494" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37899 } }
I20260619 14:57:14.351487 11688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:14.352677 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03aa90806d9b43ad847635f0360c0494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03aa90806d9b43ad847635f0360c0494" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37899 } } }
I20260619 14:57:14.352792 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:14.352993 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03aa90806d9b43ad847635f0360c0494. Latest consensus state: current_term: 1 leader_uuid: "03aa90806d9b43ad847635f0360c0494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03aa90806d9b43ad847635f0360c0494" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37899 } } }
I20260619 14:57:14.353087 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494 [sys.catalog]: This master's current role is: LEADER
W20260619 14:57:14.356073 11708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:57:14.356153 11708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:57:14.356206 11702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:14.356916 11702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:14.361608 11702 catalog_manager.cc:1347] Generated new cluster ID: 5d137b83981248a48bb2b3c7ebeb4c74
I20260619 14:57:14.361667 11702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:14.387713 11702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:14.388805 11702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:14.395988 11702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: Generated new TSK 0
I20260619 14:57:14.396623 11702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:14.420830 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37899
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:14.520145 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:14.520370 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:14.520445 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:14.524202 11714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:14.524252 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:14.524346 11714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:14.524434 11714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:14.529529 11714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:14.530383 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:14.531294 11714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:14.539779 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.539826 11714 server_base.cc:1034] running on GCE node
W20260619 14:57:14.539791 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:14.539780 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.540457 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:14.541015 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:14.542164 11714 hybrid_clock.cc:648] HybridClock initialized: now 1781881034542139 us; error 38 us; skew 500 ppm
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:14.544629 11714 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:14.545317 11714 webserver.cc:460] Webserver started at http://127.10.71.129:38005/ using document root <none> and password file <none>
I20260619 14:57:14.545784 11714 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:14.545863 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:14.546041 11714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:14.547691 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "48cc2ac064cf47d0b0291cfc0f909aee"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "eb100acc530169ce6f882b01a6d8a892"
server_key_iv: "e6f91eb6880929d4180a2e2ab5204c6e"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.548205 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "48cc2ac064cf47d0b0291cfc0f909aee"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "eb100acc530169ce6f882b01a6d8a892"
server_key_iv: "e6f91eb6880929d4180a2e2ab5204c6e"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.551353 11714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:57:14.553737 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.554939 11714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:14.555048 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "48cc2ac064cf47d0b0291cfc0f909aee"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "eb100acc530169ce6f882b01a6d8a892"
server_key_iv: "e6f91eb6880929d4180a2e2ab5204c6e"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.555145 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:14.568013 11714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:14.570408 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:14.570569 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:14.571743 11714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:14.571786 11714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.571822 11714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:14.571877 11714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:14.581812 11714 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:35131
I20260619 14:57:14.581970 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:35131 every 8 connection(s)
I20260619 14:57:14.582756 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:14.584933 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.572537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:39351 (local address 127.10.71.190:37899)
0619 14:57:14.572717 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.572722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.573360 (+ 638us) server_negotiation.cc:407] Connection header received
0619 14:57:14.574317 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.574320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.574373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.574454 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.576213 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.577037 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.578623 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.578966 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.579061 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.581544 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.581548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.581553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.581582 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.583112 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.583655 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.583658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.583659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.583757 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.584060 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.584063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.584064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.584226 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.584356 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.584736 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.584826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:57:14.585620 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.572852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.10.71.129:39351)
0619 14:57:14.573223 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:14.573264 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:14.574132 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:14.574638 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:14.574645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:14.575468 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:14.576054 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.576074 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.577179 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.577183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.578501 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.578513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.579085 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.579089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.579258 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.579743 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:14.579759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:14.581364 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:14.583253 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.583258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.583270 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.583563 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.583844 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.583849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.583852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.583978 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.584361 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:14.584366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:14.584553 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 14:57:14.584692 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:57:14.584889 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260619 14:57:14.585762 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11714
I20260619 14:57:14.585871 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:14.586114 10526 external_mini_cluster.cc:1428] Setting key c13a20e6792b43e445a2012b8cf282b8
I20260619 14:57:14.587378 11845 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37899
I20260619 14:57:14.587527 11845 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:14.587981 11845 heartbeater.cc:511] Master 127.10.71.190:37899 requested a full tablet report, sending...
I20260619 14:57:14.590240 11641 ts_manager.cc:194] Registered new tserver with Master: 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131)
I20260619 14:57:14.591688 11641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:39351
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:14.596304 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.589540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.10.71.129:33207)
0619 14:57:14.589837 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:14.589851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:14.589941 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:14.590184 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:14.590187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:14.590400 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:14.590499 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.590505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.591205 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.591208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.591761 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.591768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.592347 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.592352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.592461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.592979 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:14.592998 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:14.593428 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:14.594953 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.594957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.594959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.595160 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.595406 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.595410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.595412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.595487 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.595858 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:14.595860 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:14.595921 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 14:57:14.596018 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:57:14.596130 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260619 14:57:14.596323 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.589612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:33207 (local address 127.10.71.190:37899)
0619 14:57:14.589703 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.589707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.589931 (+ 224us) server_negotiation.cc:407] Connection header received
0619 14:57:14.590012 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.590017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.590064 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.590155 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.590603 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.591091 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.591915 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.592203 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.592339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.593563 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.593566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.593568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.593598 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.594841 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.595246 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.595250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.595252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.595324 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.595564 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.595568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.595570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.595718 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.595814 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.596064 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.596161 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:14.610594 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37899
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:14.707059 11854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:14.707249 11854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:14.707286 11854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:14.710773 11854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:14.710813 11854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:14.710878 11854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:14.710920 11854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:14.715598 11854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:14.716383 11854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:14.717186 11854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:14.725677 11862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.725734 11854 server_base.cc:1034] running on GCE node
W20260619 14:57:14.725670 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:14.725670 11859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.726244 11854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:14.726728 11854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:14.727895 11854 hybrid_clock.cc:648] HybridClock initialized: now 1781881034727883 us; error 24 us; skew 500 ppm
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:14.730571 11854 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:14.731222 11854 webserver.cc:460] Webserver started at http://127.10.71.130:41261/ using document root <none> and password file <none>
I20260619 14:57:14.731667 11854 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:14.731720 11854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:14.731861 11854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:14.733615 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "65c0dc1872f043dc91cd6718dfadff4a"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "ec5564a340fab0d773f3ab90b132674a"
server_key_iv: "d17959843ac91324ab863f331a1b2b19"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.734050 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "65c0dc1872f043dc91cd6718dfadff4a"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "ec5564a340fab0d773f3ab90b132674a"
server_key_iv: "d17959843ac91324ab863f331a1b2b19"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.737088 11854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:14.739013 11869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.739984 11854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:14.740063 11854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "65c0dc1872f043dc91cd6718dfadff4a"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "ec5564a340fab0d773f3ab90b132674a"
server_key_iv: "d17959843ac91324ab863f331a1b2b19"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.740144 11854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:14.756088 11854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:14.758791 11854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:14.758951 11854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:14.760155 11854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:14.760200 11854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.760257 11854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:14.760289 11854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:14.769882 11854 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:38337
I20260619 14:57:14.770035 11984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:38337 every 8 connection(s)
I20260619 14:57:14.770684 11854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:14.773648 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.760928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:49407 (local address 127.10.71.190:37899)
0619 14:57:14.761083 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.761088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.761846 (+ 758us) server_negotiation.cc:407] Connection header received
0619 14:57:14.762869 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.762875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.762941 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.763054 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.764707 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.765483 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.767218 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.767418 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.767494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.770259 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.770262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.770264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.770287 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.771818 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.772454 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.772458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.772459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.772528 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.772850 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.772852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.772853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.772995 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.773059 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.773457 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.773535 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:57:14.774309 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.761275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.10.71.130:49407)
0619 14:57:14.761699 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:14.761743 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:14.762649 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:14.763348 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:14.763358 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:14.763813 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:14.764548 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.764569 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.765657 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.765660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.767107 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.767116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.767556 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.767561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.767764 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.768409 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:14.768433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:14.770108 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:14.771986 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.771990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.772000 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.772264 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.772612 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.772614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.772615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.772763 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.773094 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:14.773098 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:14.773274 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 14:57:14.773407 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:57:14.773603 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260619 14:57:14.775430 11985 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37899
I20260619 14:57:14.775463 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11854
I20260619 14:57:14.775540 11985 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:14.775575 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:14.775915 10526 external_mini_cluster.cc:1428] Setting key c67f4e896ad09afd59d981ba9b184d60
I20260619 14:57:14.775928 11985 heartbeater.cc:511] Master 127.10.71.190:37899 requested a full tablet report, sending...
I20260619 14:57:14.777048 11641 ts_manager.cc:194] Registered new tserver with Master: 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337)
I20260619 14:57:14.777737 11641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:49407
I20260619 14:57:14.784994 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.777762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:36811 (local address 127.10.71.190:37899)
0619 14:57:14.777999 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.778009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.778137 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:57:14.778321 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.778342 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.778382 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.778458 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.779013 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.779649 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.780563 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.780753 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.780831 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.782042 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.782045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.782047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.782089 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.783429 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.783888 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.783890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.783891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.783954 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.784237 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.784238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.784239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.784475 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.784549 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.784828 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.784903 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260619 14:57:14.785158 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.777756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.10.71.130:36811)
0619 14:57:14.778048 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:14.778066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:14.778207 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:14.778495 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:14.778500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:14.778738 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:14.778885 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.778896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.779761 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.779763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.780442 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.780454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.780864 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.780870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.781006 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.781421 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:14.781443 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:14.781914 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:14.783544 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.783550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.783553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.783794 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.784035 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.784038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.784041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.784144 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.784567 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:14.784571 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:14.784668 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 14:57:14.784785 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:57:14.784913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:14.801123 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37899
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:14.907182 11994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:14.907403 11994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:14.907444 11994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:14.911443 11994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:14.911491 11994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:14.911580 11994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:14.911631 11994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:14.916462 11994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:14.917276 11994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:14.918143 11994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:14.926699 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:14.926715 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.926820 11994 server_base.cc:1034] running on GCE node
W20260619 14:57:14.926699 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:14.927351 11994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:14.927965 11994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:14.929181 11994 hybrid_clock.cc:648] HybridClock initialized: now 1781881034929170 us; error 23 us; skew 500 ppm
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:14.931650 11994 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:14.932509 11994 webserver.cc:460] Webserver started at http://127.10.71.131:39261/ using document root <none> and password file <none>
I20260619 14:57:14.933027 11994 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:14.933120 11994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:14.933334 11994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:14.935201 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "7800d542ddae4361861bd63641708ec8"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "def034e6ff5661bc156090c7c4cb4d14"
server_key_iv: "90499efa2452d119eaebf14550bc0810"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.935752 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "7800d542ddae4361861bd63641708ec8"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "def034e6ff5661bc156090c7c4cb4d14"
server_key_iv: "90499efa2452d119eaebf14550bc0810"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.939217 11994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:57:14.941545 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.942639 11994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:14.942792 11994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "7800d542ddae4361861bd63641708ec8"
format_stamp: "Formatted at 2026-06-19 14:57:14 on dist-test-slave-3qdh"
server_key: "def034e6ff5661bc156090c7c4cb4d14"
server_key_iv: "90499efa2452d119eaebf14550bc0810"
server_key_version: "encryptionkey@0"
I20260619 14:57:14.942894 11994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:14.966320 11994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:14.969308 11994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:14.969528 11994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:14.970911 11994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:14.970983 11994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:14.971058 11994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:14.971124 11994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:14 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:14.981804 11994 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:44235
I20260619 14:57:14.981833 12124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:44235 every 8 connection(s)
I20260619 14:57:14.982641 11994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:14.985106 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.971935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:47789 (local address 127.10.71.190:37899)
0619 14:57:14.972063 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.972066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.973016 (+ 950us) server_negotiation.cc:407] Connection header received
0619 14:57:14.974308 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.974312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.974387 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.974494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.976101 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.976741 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.978495 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.978807 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.978886 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.981710 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.981714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.981717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.981745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.983402 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.983869 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.983872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.983873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.983952 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.984179 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.984181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.984182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.984421 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.984496 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.984873 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.984965 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:57:14.985818 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.972357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.10.71.131:47789)
0619 14:57:14.972851 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:14.972891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:14.973969 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:14.974713 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:14.974726 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:14.975171 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:14.975913 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.975937 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.976870 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.976875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.978365 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.978377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.978949 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.978954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.979161 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.979775 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:14.979798 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:14.981504 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:14.983534 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.983539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.983549 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.983780 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.984011 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.984013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.984014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.984125 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.984489 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:14.984494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:14.984686 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 14:57:14.984825 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:57:14.985033 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20260619 14:57:14.986521 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 11994
I20260619 14:57:14.986642 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:14.986905 10526 external_mini_cluster.cc:1428] Setting key f4da1eccd57c4b963f4abaedeee1673e
I20260619 14:57:14.987193 12125 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37899
I20260619 14:57:14.987380 12125 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:14.988168 12125 heartbeater.cc:511] Master 127.10.71.190:37899 requested a full tablet report, sending...
I20260619 14:57:14.989244 11642 ts_manager.cc:194] Registered new tserver with Master: 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235)
I20260619 14:57:14.989807 11642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:47789
I20260619 14:57:14.995748 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.989173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.10.71.131:39589)
0619 14:57:14.989417 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:14.989428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:14.989528 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:14.989746 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:14.989749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:14.989932 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:14.990073 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.990082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.991001 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.991003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.991559 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:14.991566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.991970 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.991973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:14.992037 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.992495 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:14.992506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:14.992808 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:14.994426 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.994428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.994429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.994614 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.994857 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:14.994859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:14.994860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:14.994929 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:14.995362 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:14.995364 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:14.995430 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 14:57:14.995506 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:57:14.995618 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260619 14:57:14.995759 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:14.989212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:39589 (local address 127.10.71.190:37899)
0619 14:57:14.989323 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:14.989327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:14.989492 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:57:14.989600 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:14.989605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:14.989645 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:14.989734 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:14.990180 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.990872 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.991679 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:14.991879 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:14.991955 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:14.992926 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:14.992929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:14.992931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:14.992951 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:14.994327 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.994699 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.994701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.994703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.994779 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:14.995081 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:14.995086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:14.995087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:14.995276 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:14.995380 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:14.995568 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 14:57:14.995658 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:57:15.000324 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[11597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:15.025561 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.017590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.10.71.190:37899)
0619 14:57:15.017723 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.017726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.017737 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:15.017772 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.017774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.017812 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.017905 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:15.018624 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.019124 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.019987 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.020211 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.020352 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.022653 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:15.022655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:15.022657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:15.022680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:15.023981 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.024515 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.024524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.024527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.024608 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.024846 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.024848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.024849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.025014 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:15.025142 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.025344 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.025435 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:57:15.028105 11642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:15.030820 11642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:15.044615 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.039684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.10.71.130:38337)
0619 14:57:15.040097 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.040108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.040431 (+ 323us) server_negotiation.cc:407] Connection header received
0619 14:57:15.040544 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.040547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.040752 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.040886 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.041042 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.041938 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.043361 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.043961 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.044079 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.044226 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.044392 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.044481 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":108,"threads_started":1}
I20260619 14:57:15.044605 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.039268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:38337 (local address 127.0.0.1:48804)
0619 14:57:15.040326 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.040339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.040432 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.040860 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.040862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.040877 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.040956 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.040962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.042045 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.042048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.043235 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.043243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.044084 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.044087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.044183 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.044219 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.044356 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.044473 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":17,"thread_start_us":86,"threads_started":1}
I20260619 14:57:15.044755 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.039041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:44235 (local address 127.0.0.1:36746)
0619 14:57:15.039364 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.039375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.039445 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.040601 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.040604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.040620 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.040700 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.040723 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.041891 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.041895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.043209 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.043220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.044247 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.044251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.044398 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.044420 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.044482 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.044548 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":84,"thread_start_us":95,"threads_started":1}
I20260619 14:57:15.044905 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.038609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:35131 (local address 127.0.0.1:42644)
0619 14:57:15.038951 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.039000 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.039119 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.039766 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.039773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.039800 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.039912 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.039918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.041546 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.041550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.043137 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.043146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.044247 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.044251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.044642 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.044664 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.044726 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.044790 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":335,"thread_start_us":60,"threads_started":1}
I20260619 14:57:15.045212 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.038758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.10.71.129:35131)
0619 14:57:15.039222 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.039235 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.039257 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:57:15.039345 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.039350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.039595 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.040085 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.040217 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.041411 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.043250 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.044036 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.044408 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.044565 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.044993 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.045079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":106,"threads_started":1}
I20260619 14:57:15.045442 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.039685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.10.71.131:44235)
0619 14:57:15.040099 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.040111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.040132 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:57:15.040204 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.040209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.040419 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.040618 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.040829 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.041785 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.043330 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.044122 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.045031 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.045187 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.045236 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.045320 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":97,"threads_started":1}
I20260619 14:57:15.047914 12059 tablet_service.cc:1467] Processing CreateTablet for tablet 713fbb89e2bb44879562380dfd555fc1 (DEFAULT_TABLE table=test-table [id=065797ca24894e9d8c13165b02057166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:15.047833 11779 tablet_service.cc:1467] Processing CreateTablet for tablet 713fbb89e2bb44879562380dfd555fc1 (DEFAULT_TABLE table=test-table [id=065797ca24894e9d8c13165b02057166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:15.047833 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 713fbb89e2bb44879562380dfd555fc1 (DEFAULT_TABLE table=test-table [id=065797ca24894e9d8c13165b02057166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:15.048776 12059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713fbb89e2bb44879562380dfd555fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.048790 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713fbb89e2bb44879562380dfd555fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.048790 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713fbb89e2bb44879562380dfd555fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.056627 12149 tablet_bootstrap.cc:492] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Bootstrap starting.
I20260619 14:57:15.058552 12149 tablet_bootstrap.cc:654] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.059326 12149 log.cc:824] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:15.059899 12148 tablet_bootstrap.cc:492] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Bootstrap starting.
I20260619 14:57:15.061147 12149 tablet_bootstrap.cc:492] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: No bootstrap required, opened a new log
I20260619 14:57:15.061336 12151 tablet_bootstrap.cc:492] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Bootstrap starting.
I20260619 14:57:15.061446 12149 ts_tablet_manager.cc:1397] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:15.062408 12148 tablet_bootstrap.cc:654] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.063118 12148 log.cc:824] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:15.063218 12151 tablet_bootstrap.cc:654] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.063974 12151 log.cc:824] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:15.064821 12148 tablet_bootstrap.cc:492] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: No bootstrap required, opened a new log
I20260619 14:57:15.065001 12148 ts_tablet_manager.cc:1397] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:15.065325 12149 raft_consensus.cc:348] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.065563 12149 raft_consensus.cc:374] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.065620 12149 raft_consensus.cc:729] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c0dc1872f043dc91cd6718dfadff4a, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.065716 12151 tablet_bootstrap.cc:492] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: No bootstrap required, opened a new log
I20260619 14:57:15.065863 12151 ts_tablet_manager.cc:1397] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:15.066054 12149 consensus_queue.cc:260] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.066679 11985 heartbeater.cc:503] Master 127.10.71.190:37899 was elected leader, sending a full tablet report...
I20260619 14:57:15.066987 12149 ts_tablet_manager.cc:1428] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:15.068478 12148 raft_consensus.cc:348] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.068732 12148 raft_consensus.cc:374] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.068797 12148 raft_consensus.cc:729] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc2ac064cf47d0b0291cfc0f909aee, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.069199 12148 consensus_queue.cc:260] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.069228 12151 raft_consensus.cc:348] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.069404 12151 raft_consensus.cc:374] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.069439 12151 raft_consensus.cc:729] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7800d542ddae4361861bd63641708ec8, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.069814 11845 heartbeater.cc:503] Master 127.10.71.190:37899 was elected leader, sending a full tablet report...
I20260619 14:57:15.069856 12151 consensus_queue.cc:260] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.070152 12148 ts_tablet_manager.cc:1428] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:15.070744 12151 ts_tablet_manager.cc:1428] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:15.071033 12125 heartbeater.cc:503] Master 127.10.71.190:37899 was elected leader, sending a full tablet report...
W20260619 14:57:15.083861 11846 tablet.cc:2369] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:15.088406 12155 raft_consensus.cc:482] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:15.088563 12155 raft_consensus.cc:504] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.089564 12155 leader_election.cc:288] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337), 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235)
I20260619 14:57:15.093621 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.089825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:44903 (local address 127.10.71.130:38337)
0619 14:57:15.089956 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.089960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.090035 (+ 75us) server_negotiation.cc:407] Connection header received
0619 14:57:15.090258 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.090262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.090311 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.090381 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.090794 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.091326 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.092579 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.092966 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.093102 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.093156 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.093425 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.093493 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:15.093628 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.090001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:59681 (local address 127.10.71.131:44235)
0619 14:57:15.090114 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.090118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.090257 (+ 139us) server_negotiation.cc:407] Connection header received
0619 14:57:15.090367 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.090373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.090421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.090501 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.090874 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.091362 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.092554 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.092968 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.093140 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.093197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.093422 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.093497 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:57:15.093653 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.089790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:44235 (local address 127.10.71.129:59681)
0619 14:57:15.090144 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.090155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.090267 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.090531 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.090534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.090547 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.090758 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.090766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.091495 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.091497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.092429 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.092438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.093137 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.093140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.093270 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.093319 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.093404 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.093490 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":103,"thread_start_us":111,"threads_started":1}
I20260619 14:57:15.093688 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.089719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:38337 (local address 127.10.71.129:44903)
0619 14:57:15.089945 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.089961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.090124 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.090404 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.090407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.090426 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.090580 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.090592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.091447 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.091453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.092433 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.092443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.093124 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.093128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.093284 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.093317 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.093410 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.093510 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":58,"thread_start_us":81,"threads_started":1}
I20260619 14:57:15.094159 12079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713fbb89e2bb44879562380dfd555fc1" candidate_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7800d542ddae4361861bd63641708ec8" is_pre_election: true
I20260619 14:57:15.094197 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713fbb89e2bb44879562380dfd555fc1" candidate_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c0dc1872f043dc91cd6718dfadff4a" is_pre_election: true
I20260619 14:57:15.094439 11939 raft_consensus.cc:2454] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc2ac064cf47d0b0291cfc0f909aee in term 0.
I20260619 14:57:15.094444 12079 raft_consensus.cc:2454] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc2ac064cf47d0b0291cfc0f909aee in term 0.
I20260619 14:57:15.094830 11733 leader_election.cc:302] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48cc2ac064cf47d0b0291cfc0f909aee, 7800d542ddae4361861bd63641708ec8; no voters:
I20260619 14:57:15.095039 12155 raft_consensus.cc:2781] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:15.095124 12155 raft_consensus.cc:482] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:15.095158 12155 raft_consensus.cc:3037] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.096392 12155 raft_consensus.cc:504] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.096768 12155 leader_election.cc:288] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [CANDIDATE]: Term 1 election: Requested vote from peers 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337), 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235)
I20260619 14:57:15.097024 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713fbb89e2bb44879562380dfd555fc1" candidate_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c0dc1872f043dc91cd6718dfadff4a"
I20260619 14:57:15.097148 11939 raft_consensus.cc:3037] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.097152 12079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713fbb89e2bb44879562380dfd555fc1" candidate_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7800d542ddae4361861bd63641708ec8"
I20260619 14:57:15.097321 12079 raft_consensus.cc:3037] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.098254 12079 raft_consensus.cc:2454] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc2ac064cf47d0b0291cfc0f909aee in term 1.
I20260619 14:57:15.098598 11733 leader_election.cc:302] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48cc2ac064cf47d0b0291cfc0f909aee, 7800d542ddae4361861bd63641708ec8; no voters:
I20260619 14:57:15.098665 11939 raft_consensus.cc:2454] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc2ac064cf47d0b0291cfc0f909aee in term 1.
I20260619 14:57:15.098841 12155 raft_consensus.cc:2781] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:15.098997 12155 raft_consensus.cc:686] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 LEADER]: Becoming Leader. State: Replica: 48cc2ac064cf47d0b0291cfc0f909aee, State: Running, Role: LEADER
I20260619 14:57:15.099303 12155 consensus_queue.cc:237] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.102037 11642 catalog_manager.cc:5495] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee reported cstate change: term changed from 0 to 1, leader changed from <none> to 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129). New cstate: current_term: 1 leader_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:15.126431 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.122544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.10.71.129:35131)
0619 14:57:15.122668 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.122674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.122732 (+ 58us) server_negotiation.cc:407] Connection header received
0619 14:57:15.123285 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.123288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.123332 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.123432 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:15.123704 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.124160 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.125082 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.125289 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.125388 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.125708 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.126053 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.126172 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.126257 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.126306 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:15.131309 12079 raft_consensus.cc:1261] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Refusing update from remote peer 48cc2ac064cf47d0b0291cfc0f909aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:15.131316 11939 raft_consensus.cc:1261] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Refusing update from remote peer 48cc2ac064cf47d0b0291cfc0f909aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:15.131970 12155 consensus_queue.cc:1035] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:15.132167 12159 consensus_queue.cc:1035] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:15.137845 12162 mvcc.cc:205] Tried to move back new op lower bound from 7298584719892090880 to 7298584719768633344. Current Snapshot: MvccSnapshot[applied={T|T < 7298584719892090880}]
I20260619 14:57:15.139420 12163 mvcc.cc:205] Tried to move back new op lower bound from 7298584719892090880 to 7298584719768633344. Current Snapshot: MvccSnapshot[applied={T|T < 7298584719892090880}]
I20260619 14:57:15.140892 12165 mvcc.cc:205] Tried to move back new op lower bound from 7298584719892090880 to 7298584719768633344. Current Snapshot: MvccSnapshot[applied={T|T < 7298584719892090880}]
I20260619 14:57:15.150454 11642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:15.150588 11642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:15.152984 11642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: Sending DeleteTablet for 3 replicas of tablet 713fbb89e2bb44879562380dfd555fc1
I20260619 14:57:15.153709 12059 tablet_service.cc:1514] Processing DeleteTablet for tablet 713fbb89e2bb44879562380dfd555fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:15 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:36746
I20260619 14:57:15.153785 11779 tablet_service.cc:1514] Processing DeleteTablet for tablet 713fbb89e2bb44879562380dfd555fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:15 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 14:57:15.154157 12171 tablet_replica.cc:330] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: stopping tablet replica
I20260619 14:57:15.154139 11919 tablet_service.cc:1514] Processing DeleteTablet for tablet 713fbb89e2bb44879562380dfd555fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:15 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:48804
I20260619 14:57:15.154446 12171 raft_consensus.cc:2229] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:15.154606 12173 tablet_replica.cc:330] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: stopping tablet replica
I20260619 14:57:15.154716 12171 raft_consensus.cc:2258] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:15.154934 12173 raft_consensus.cc:2229] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:15.155231 12173 raft_consensus.cc:2258] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:15.155328 11642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:15.155678 11642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:15.156128 12171 ts_tablet_manager.cc:1905] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:15.156598 12173 ts_tablet_manager.cc:1905] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:15.158995 12173 ts_tablet_manager.cc:1918] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:15.159068 12171 ts_tablet_manager.cc:1918] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:15.159109 12173 log.cc:1196] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/713fbb89e2bb44879562380dfd555fc1
I20260619 14:57:15.159196 12171 log.cc:1196] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/713fbb89e2bb44879562380dfd555fc1
I20260619 14:57:15.159413 12173 ts_tablet_manager.cc:1939] T 713fbb89e2bb44879562380dfd555fc1 P 65c0dc1872f043dc91cd6718dfadff4a: Deleting consensus metadata
I20260619 14:57:15.159523 12171 ts_tablet_manager.cc:1939] T 713fbb89e2bb44879562380dfd555fc1 P 7800d542ddae4361861bd63641708ec8: Deleting consensus metadata
I20260619 14:57:15.160521 11626 catalog_manager.cc:4841] TS 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337): tablet 713fbb89e2bb44879562380dfd555fc1 (table test-table [id=065797ca24894e9d8c13165b02057166]) successfully deleted
I20260619 14:57:15.160640 12172 tablet_replica.cc:330] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: stopping tablet replica
I20260619 14:57:15.160846 11629 catalog_manager.cc:4841] TS 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235): tablet 713fbb89e2bb44879562380dfd555fc1 (table test-table [id=065797ca24894e9d8c13165b02057166]) successfully deleted
I20260619 14:57:15.160997 11779 tablet_service.cc:1467] Processing CreateTablet for tablet 462182f031184f19b8f74a6d090545bd (DEFAULT_TABLE table=test-table [id=4882946524024617b7b0904a79336c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:15.161217 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462182f031184f19b8f74a6d090545bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.161267 12059 tablet_service.cc:1467] Processing CreateTablet for tablet 462182f031184f19b8f74a6d090545bd (DEFAULT_TABLE table=test-table [id=4882946524024617b7b0904a79336c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:15.161405 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 462182f031184f19b8f74a6d090545bd (DEFAULT_TABLE table=test-table [id=4882946524024617b7b0904a79336c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:15.161463 12059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462182f031184f19b8f74a6d090545bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.161616 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462182f031184f19b8f74a6d090545bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.163480 12148 tablet_bootstrap.cc:492] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Bootstrap starting.
I20260619 14:57:15.163578 12149 tablet_bootstrap.cc:492] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Bootstrap starting.
I20260619 14:57:15.163578 12151 tablet_bootstrap.cc:492] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Bootstrap starting.
I20260619 14:57:15.164402 12148 tablet_bootstrap.cc:654] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.164485 12151 tablet_bootstrap.cc:654] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.164485 12149 tablet_bootstrap.cc:654] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.165309 12172 raft_consensus.cc:2229] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:15.165663 12172 raft_consensus.cc:2258] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:15.166232 12149 tablet_bootstrap.cc:492] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: No bootstrap required, opened a new log
I20260619 14:57:15.166316 12149 ts_tablet_manager.cc:1397] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:57:15.166477 12148 tablet_bootstrap.cc:492] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: No bootstrap required, opened a new log
I20260619 14:57:15.166543 12148 ts_tablet_manager.cc:1397] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 14:57:15.166702 12151 tablet_bootstrap.cc:492] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: No bootstrap required, opened a new log
I20260619 14:57:15.166774 12151 ts_tablet_manager.cc:1397] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:57:15.166769 12149 raft_consensus.cc:348] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.166865 12149 raft_consensus.cc:374] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.166903 12149 raft_consensus.cc:729] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c0dc1872f043dc91cd6718dfadff4a, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.166937 12148 raft_consensus.cc:348] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.167023 12148 raft_consensus.cc:374] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.167052 12148 raft_consensus.cc:729] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc2ac064cf47d0b0291cfc0f909aee, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.167018 12149 consensus_queue.cc:260] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.167152 12148 consensus_queue.cc:260] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.167200 12151 raft_consensus.cc:348] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.167263 12149 ts_tablet_manager.cc:1428] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:15.167296 12151 raft_consensus.cc:374] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.167328 12151 raft_consensus.cc:729] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7800d542ddae4361861bd63641708ec8, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.167356 12148 ts_tablet_manager.cc:1428] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:15.167446 12151 consensus_queue.cc:260] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.167677 12151 ts_tablet_manager.cc:1428] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:15.169382 12172 ts_tablet_manager.cc:1905] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:15.171658 12172 ts_tablet_manager.cc:1918] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:15.171729 12172 log.cc:1196] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/713fbb89e2bb44879562380dfd555fc1
I20260619 14:57:15.171998 12172 ts_tablet_manager.cc:1939] T 713fbb89e2bb44879562380dfd555fc1 P 48cc2ac064cf47d0b0291cfc0f909aee: Deleting consensus metadata
I20260619 14:57:15.172891 11626 catalog_manager.cc:4841] TS 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131): tablet 713fbb89e2bb44879562380dfd555fc1 (table test-table [id=065797ca24894e9d8c13165b02057166]) successfully deleted
W20260619 14:57:15.233747 12126 tablet.cc:2369] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:57:15.272441 11986 tablet.cc:2369] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:15.345687 12154 raft_consensus.cc:482] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:15.345810 12154 raft_consensus.cc:504] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.346954 12154 leader_election.cc:288] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131), 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235)
I20260619 14:57:15.350559 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.347449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:60045 (local address 127.10.71.131:44235)
0619 14:57:15.347620 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.347623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.347634 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:15.347763 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.347766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.347819 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.347893 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.348147 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.348686 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.349674 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.350037 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.350153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.350204 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.350373 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.350440 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:57:15.350585 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.347211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:45335 (local address 127.10.71.129:35131)
0619 14:57:15.347350 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.347355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.347467 (+ 112us) server_negotiation.cc:407] Connection header received
0619 14:57:15.347616 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.347619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.347670 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.347751 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.348042 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.348587 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.349675 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.350050 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.350182 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.350233 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.350417 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.350486 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:57:15.350575 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.347209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:44235 (local address 127.10.71.130:60045)
0619 14:57:15.347530 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.347545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.347638 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.347920 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.347923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.347936 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.348043 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.348049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.348791 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.348793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.349548 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.349556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.350153 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.350157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.350243 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.350270 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.350336 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.350417 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":68,"threads_started":1}
I20260619 14:57:15.350581 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.347108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:35131 (local address 127.10.71.130:45335)
0619 14:57:15.347375 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.347386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.347508 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.347767 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.347770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.347786 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.347932 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.347939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.348684 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.348687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.349555 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.349563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.350182 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.350186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.350283 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.350299 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.350389 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.350448 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":39,"thread_start_us":67,"threads_started":1}
I20260619 14:57:15.351059 12078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462182f031184f19b8f74a6d090545bd" candidate_uuid: "65c0dc1872f043dc91cd6718dfadff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7800d542ddae4361861bd63641708ec8" is_pre_election: true
I20260619 14:57:15.351205 12078 raft_consensus.cc:2454] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c0dc1872f043dc91cd6718dfadff4a in term 0.
I20260619 14:57:15.351421 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462182f031184f19b8f74a6d090545bd" candidate_uuid: "65c0dc1872f043dc91cd6718dfadff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" is_pre_election: true
I20260619 14:57:15.351624 11799 raft_consensus.cc:2454] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c0dc1872f043dc91cd6718dfadff4a in term 0.
I20260619 14:57:15.352026 11870 leader_election.cc:302] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48cc2ac064cf47d0b0291cfc0f909aee, 65c0dc1872f043dc91cd6718dfadff4a; no voters:
I20260619 14:57:15.352445 12154 raft_consensus.cc:2781] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:15.352516 12154 raft_consensus.cc:482] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:15.352569 12154 raft_consensus.cc:3037] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.353718 12154 raft_consensus.cc:504] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.354058 12154 leader_election.cc:288] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [CANDIDATE]: Term 1 election: Requested vote from peers 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131), 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235)
I20260619 14:57:15.354342 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462182f031184f19b8f74a6d090545bd" candidate_uuid: "65c0dc1872f043dc91cd6718dfadff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc2ac064cf47d0b0291cfc0f909aee"
I20260619 14:57:15.354342 12078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462182f031184f19b8f74a6d090545bd" candidate_uuid: "65c0dc1872f043dc91cd6718dfadff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7800d542ddae4361861bd63641708ec8"
I20260619 14:57:15.354460 11799 raft_consensus.cc:3037] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.354466 12078 raft_consensus.cc:3037] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.355590 12078 raft_consensus.cc:2454] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c0dc1872f043dc91cd6718dfadff4a in term 1.
I20260619 14:57:15.355616 11799 raft_consensus.cc:2454] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c0dc1872f043dc91cd6718dfadff4a in term 1.
I20260619 14:57:15.355873 11870 leader_election.cc:302] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48cc2ac064cf47d0b0291cfc0f909aee, 65c0dc1872f043dc91cd6718dfadff4a; no voters:
I20260619 14:57:15.356047 12154 raft_consensus.cc:2781] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:15.356230 12154 raft_consensus.cc:686] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 1 LEADER]: Becoming Leader. State: Replica: 65c0dc1872f043dc91cd6718dfadff4a, State: Running, Role: LEADER
I20260619 14:57:15.356472 12154 consensus_queue.cc:237] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.358538 11642 catalog_manager.cc:5495] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130). New cstate: current_term: 1 leader_uuid: "65c0dc1872f043dc91cd6718dfadff4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881034, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:15.436812 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.429725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37899 (local address 127.0.0.1:51050)
0619 14:57:15.430024 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.430034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.430122 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.430328 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.430330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.430686 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:15.430767 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.430773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.431449 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.431451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.432013 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.432020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.432462 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.432464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.432523 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.433005 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:15.433015 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:15.434254 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:15.435603 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:15.435610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:15.435614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:15.435801 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:15.436006 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:15.436008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:15.436009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:15.436077 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:15.436417 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:15.436422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:15.436510 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.436607 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.436695 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":81,"threads_started":1}
I20260619 14:57:15.436970 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.429813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.10.71.190:37899)
0619 14:57:15.429921 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.429924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.430117 (+ 193us) server_negotiation.cc:407] Connection header received
0619 14:57:15.430195 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.430197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.430233 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.430292 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:15.430860 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.431358 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.432121 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.432372 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.432446 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.434364 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:15.434367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:15.434369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:15.434387 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:15.435510 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.435876 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.435878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.435879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.435939 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.436158 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.436160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.436161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.436307 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:15.436392 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.436708 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.436800 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:57:15.440122 11642 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:51050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:57:15.440804 11642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:15.445268 11779 tablet_service.cc:1467] Processing CreateTablet for tablet ceeaca753dde40e0b3f99eced7675645 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e4bab15b1754d21826512802172a2fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:15.445344 11919 tablet_service.cc:1467] Processing CreateTablet for tablet ceeaca753dde40e0b3f99eced7675645 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e4bab15b1754d21826512802172a2fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:15.445518 12059 tablet_service.cc:1467] Processing CreateTablet for tablet ceeaca753dde40e0b3f99eced7675645 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e4bab15b1754d21826512802172a2fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:15.445667 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeaca753dde40e0b3f99eced7675645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.445696 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeaca753dde40e0b3f99eced7675645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.445820 12059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeaca753dde40e0b3f99eced7675645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:15.448057 12149 tablet_bootstrap.cc:492] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: Bootstrap starting.
I20260619 14:57:15.448262 12148 tablet_bootstrap.cc:492] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: Bootstrap starting.
I20260619 14:57:15.448535 12151 tablet_bootstrap.cc:492] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: Bootstrap starting.
I20260619 14:57:15.448997 12148 tablet_bootstrap.cc:654] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.448997 12149 tablet_bootstrap.cc:654] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.449267 12151 tablet_bootstrap.cc:654] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:15.450109 12148 tablet_bootstrap.cc:492] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: No bootstrap required, opened a new log
I20260619 14:57:15.450136 12149 tablet_bootstrap.cc:492] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: No bootstrap required, opened a new log
I20260619 14:57:15.450204 12148 ts_tablet_manager.cc:1397] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 14:57:15.450212 12149 ts_tablet_manager.cc:1397] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:15.450552 12148 raft_consensus.cc:348] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.450660 12148 raft_consensus.cc:374] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.450691 12148 raft_consensus.cc:729] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc2ac064cf47d0b0291cfc0f909aee, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.450630 12149 raft_consensus.cc:348] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.450747 12149 raft_consensus.cc:374] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.450783 12149 raft_consensus.cc:729] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c0dc1872f043dc91cd6718dfadff4a, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.450834 12151 tablet_bootstrap.cc:492] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: No bootstrap required, opened a new log
I20260619 14:57:15.450802 12148 consensus_queue.cc:260] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.450903 12151 ts_tablet_manager.cc:1397] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:15.450904 12149 consensus_queue.cc:260] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.451063 12148 ts_tablet_manager.cc:1428] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:15.451172 12149 ts_tablet_manager.cc:1428] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:15.451268 12159 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.451293 12151 raft_consensus.cc:348] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.451385 12151 raft_consensus.cc:374] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:15.451395 12159 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.451346 12154 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.451422 12151 raft_consensus.cc:729] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7800d542ddae4361861bd63641708ec8, State: Initialized, Role: FOLLOWER
I20260619 14:57:15.451459 12154 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.451531 12151 consensus_queue.cc:260] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.451798 12151 ts_tablet_manager.cc:1428] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:15.451802 12156 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.452009 12156 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.609046 12156 raft_consensus.cc:482] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:15.609156 12156 raft_consensus.cc:504] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.610049 12156 leader_election.cc:288] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131), 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337)
I20260619 14:57:15.613847 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.610336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:40513 (local address 127.10.71.129:35131)
0619 14:57:15.610453 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.610458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.610606 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:57:15.610768 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.610772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.610815 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.610876 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.611183 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.611768 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.612907 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.613311 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.613393 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.613427 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.613669 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.613746 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:57:15.613881 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.610292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:35131 (local address 127.10.71.131:40513)
0619 14:57:15.610497 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.610508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.610628 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.610911 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.610914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.610934 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.611061 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.611067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.611874 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.611877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.612798 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.612807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.613428 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.613432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.613533 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.613572 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.613644 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.613716 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"spinlock_wait_cycles":6912,"thread_start_us":72,"threads_started":1}
I20260619 14:57:15.613919 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.610646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:38337 (local address 127.10.71.131:36095)
0619 14:57:15.610884 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.610895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.610960 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.611180 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.611183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.611197 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:15.611307 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.611312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.612061 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.612064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.612900 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.612908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.613573 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.613576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.613663 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.613684 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.613748 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.613803 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":62,"threads_started":1}
I20260619 14:57:15.613972 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.610741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:36095 (local address 127.10.71.130:38337)
0619 14:57:15.610945 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.610949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.610962 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:15.611033 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.611037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.611087 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.611164 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:15.611404 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.611954 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.613026 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.613470 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.613590 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.613653 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.613776 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.613844 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260619 14:57:15.614348 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeaca753dde40e0b3f99eced7675645" candidate_uuid: "7800d542ddae4361861bd63641708ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c0dc1872f043dc91cd6718dfadff4a" is_pre_election: true
I20260619 14:57:15.614426 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeaca753dde40e0b3f99eced7675645" candidate_uuid: "7800d542ddae4361861bd63641708ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" is_pre_election: true
I20260619 14:57:15.614492 11939 raft_consensus.cc:2454] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7800d542ddae4361861bd63641708ec8 in term 0.
I20260619 14:57:15.614585 11799 raft_consensus.cc:2454] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7800d542ddae4361861bd63641708ec8 in term 0.
I20260619 14:57:15.614776 12010 leader_election.cc:302] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c0dc1872f043dc91cd6718dfadff4a, 7800d542ddae4361861bd63641708ec8; no voters:
I20260619 14:57:15.615018 12156 raft_consensus.cc:2781] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:15.615088 12156 raft_consensus.cc:482] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:15.615140 12156 raft_consensus.cc:3037] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.616166 12156 raft_consensus.cc:504] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.616539 12156 leader_election.cc:288] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131), 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337)
I20260619 14:57:15.616847 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeaca753dde40e0b3f99eced7675645" candidate_uuid: "7800d542ddae4361861bd63641708ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc2ac064cf47d0b0291cfc0f909aee"
I20260619 14:57:15.616933 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeaca753dde40e0b3f99eced7675645" candidate_uuid: "7800d542ddae4361861bd63641708ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c0dc1872f043dc91cd6718dfadff4a"
I20260619 14:57:15.616998 11799 raft_consensus.cc:3037] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.617043 11939 raft_consensus.cc:3037] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:15.617847 11939 raft_consensus.cc:2454] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7800d542ddae4361861bd63641708ec8 in term 1.
I20260619 14:57:15.617857 11799 raft_consensus.cc:2454] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7800d542ddae4361861bd63641708ec8 in term 1.
I20260619 14:57:15.618089 12010 leader_election.cc:302] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c0dc1872f043dc91cd6718dfadff4a, 7800d542ddae4361861bd63641708ec8; no voters:
I20260619 14:57:15.618260 12156 raft_consensus.cc:2781] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:15.618443 12156 raft_consensus.cc:686] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [term 1 LEADER]: Becoming Leader. State: Replica: 7800d542ddae4361861bd63641708ec8, State: Running, Role: LEADER
I20260619 14:57:15.618673 12156 consensus_queue.cc:237] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } }
I20260619 14:57:15.619681 12193 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: TxnStatusTablet state changed. Reason: New leader 7800d542ddae4361861bd63641708ec8. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.619779 12193 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:15.620222 12195 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:57:15.620395 12195 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:57:15.620798 11639 catalog_manager.cc:5495] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7800d542ddae4361861bd63641708ec8 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } health_report { overall_health: HEALTHY } } }
I20260619 14:57:15.643083 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.639400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.10.71.131:44235)
0619 14:57:15.639514 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.639517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.639530 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:15.639710 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.639712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.639780 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.639869 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:15.640093 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.640560 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.641692 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.641903 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.641987 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.642301 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.642750 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.642851 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.642959 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.642999 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:15.643190 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.639347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:44235 (local address 127.0.0.1:36754)
0619 14:57:15.639471 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.639484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.639569 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.639897 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.639900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.639907 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:15.639998 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.640003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.640653 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.640657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.641533 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.641542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.642011 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.642015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.642124 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.642171 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.642857 (+ 686us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.642871 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.642943 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.643022 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:57:15.645697 11939 raft_consensus.cc:1261] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Refusing update from remote peer 7800d542ddae4361861bd63641708ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:15.645747 11799 raft_consensus.cc:1261] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Refusing update from remote peer 7800d542ddae4361861bd63641708ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:15.646101 12193 consensus_queue.cc:1035] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:15.646251 12156 consensus_queue.cc:1035] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:15.647632 12154 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: TxnStatusTablet state changed. Reason: New leader 7800d542ddae4361861bd63641708ec8. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.647728 12154 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.648428 12159 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: TxnStatusTablet state changed. Reason: New leader 7800d542ddae4361861bd63641708ec8. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.648546 12159 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.650277 12156 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.650368 12156 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:15.651382 12156 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.651474 12156 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 7800d542ddae4361861bd63641708ec8: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:15.652561 12159 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.652657 12159 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 48cc2ac064cf47d0b0291cfc0f909aee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.653659 12154 tablet_replica.cc:439] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7800d542ddae4361861bd63641708ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 } } peers { permanent_uuid: "65c0dc1872f043dc91cd6718dfadff4a" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38337 } } peers { permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 } } }
I20260619 14:57:15.653748 12154 tablet_replica.cc:442] T ceeaca753dde40e0b3f99eced7675645 P 65c0dc1872f043dc91cd6718dfadff4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:15.663745 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.655471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.10.71.130:38337)
0619 14:57:15.655583 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.655591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.660544 (+ 4953us) server_negotiation.cc:407] Connection header received
0619 14:57:15.660656 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.660658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.660698 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.660767 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:15.661002 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.661417 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.662598 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.662809 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.662890 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.663130 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.663425 (+ 295us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.663497 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.663587 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.663626 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:57:15.672607 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.669450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:39179 (local address 127.10.71.131:44235)
0619 14:57:15.669532 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.669535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.669712 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:57:15.669782 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.669785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.669824 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.669896 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:15.670093 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.670756 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.671466 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.671809 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.671903 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.672125 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.672192 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.672261 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.672460 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.672502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:57:15.672622 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.669376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:44235 (local address 127.10.71.130:39179)
0619 14:57:15.669620 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.669631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.669722 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.669914 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.669915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.669921 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:15.669994 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.669997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.670842 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.670844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.671373 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.671380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.671909 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.671911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.671975 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.672008 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.672358 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.672365 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.672432 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.672492 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260619 14:57:15.678687 11799 raft_consensus.cc:1261] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Refusing update from remote peer 65c0dc1872f043dc91cd6718dfadff4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:15.678818 12078 raft_consensus.cc:1261] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Refusing update from remote peer 65c0dc1872f043dc91cd6718dfadff4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:15.679100 12179 consensus_queue.cc:1035] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc2ac064cf47d0b0291cfc0f909aee" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:15.679291 12154 consensus_queue.cc:1035] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7800d542ddae4361861bd63641708ec8" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:15.700278 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.697049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:36997 (local address 127.10.71.130:38337)
0619 14:57:15.697156 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.697160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.697341 (+ 181us) server_negotiation.cc:407] Connection header received
0619 14:57:15.697440 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.697443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.697475 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.697523 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:15.697725 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.698342 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.699394 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.699679 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.699769 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.699963 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.700022 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.700088 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.700144 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.700182 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:57:15.700356 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.697000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:38337 (local address 127.10.71.131:36997)
0619 14:57:15.697267 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:15.697279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:15.697365 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:15.697546 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:15.697549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:15.697554 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:15.697650 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.697655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.698460 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.698465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.699297 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:15.699306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.699782 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.699784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:15.699837 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.699868 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:15.700090 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:15.700094 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:57:15.700134 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 14:57:15.700208 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":111,"threads_started":1}
I20260619 14:57:15.721055 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:15.721196 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:15.723245 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03aa90806d9b43ad847635f0360c0494: Sending DeleteTablet for 3 replicas of tablet 462182f031184f19b8f74a6d090545bd
I20260619 14:57:15.723857 11779 tablet_service.cc:1514] Processing DeleteTablet for tablet 462182f031184f19b8f74a6d090545bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:15 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:42644
I20260619 14:57:15.723938 11919 tablet_service.cc:1514] Processing DeleteTablet for tablet 462182f031184f19b8f74a6d090545bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:15 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:48804
I20260619 14:57:15.724221 12219 tablet_replica.cc:330] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: stopping tablet replica
I20260619 14:57:15.724404 12219 raft_consensus.cc:2229] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:15.724543 12219 raft_consensus.cc:2258] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:15.724830 12059 tablet_service.cc:1514] Processing DeleteTablet for tablet 462182f031184f19b8f74a6d090545bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:15 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:36746
I20260619 14:57:15.725129 12222 tablet_replica.cc:330] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: stopping tablet replica
I20260619 14:57:15.725241 12222 raft_consensus.cc:2229] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:15.725342 12222 raft_consensus.cc:2258] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:15.725348 12219 ts_tablet_manager.cc:1905] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:15.725909 12222 ts_tablet_manager.cc:1905] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:15.728369 12219 ts_tablet_manager.cc:1918] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:15.728497 12219 log.cc:1196] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/462182f031184f19b8f74a6d090545bd
I20260619 14:57:15.728576 12222 ts_tablet_manager.cc:1918] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:15.728667 12222 log.cc:1196] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/462182f031184f19b8f74a6d090545bd
I20260619 14:57:15.729102 12219 ts_tablet_manager.cc:1939] T 462182f031184f19b8f74a6d090545bd P 48cc2ac064cf47d0b0291cfc0f909aee: Deleting consensus metadata
I20260619 14:57:15.729738 12222 ts_tablet_manager.cc:1939] T 462182f031184f19b8f74a6d090545bd P 7800d542ddae4361861bd63641708ec8: Deleting consensus metadata
I20260619 14:57:15.729715 11626 catalog_manager.cc:4841] TS 48cc2ac064cf47d0b0291cfc0f909aee (127.10.71.129:35131): tablet 462182f031184f19b8f74a6d090545bd (table test-table [id=4882946524024617b7b0904a79336c6a]) successfully deleted
I20260619 14:57:15.730341 11629 catalog_manager.cc:4841] TS 7800d542ddae4361861bd63641708ec8 (127.10.71.131:44235): tablet 462182f031184f19b8f74a6d090545bd (table test-table [id=4882946524024617b7b0904a79336c6a]) successfully deleted
I20260619 14:57:15.732481 12220 tablet_replica.cc:330] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: stopping tablet replica
I20260619 14:57:15.732597 12220 raft_consensus.cc:2229] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:15.733994 12220 raft_consensus.cc:2258] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:15.735618 12220 ts_tablet_manager.cc:1905] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[11597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:15.739113 12220 ts_tablet_manager.cc:1918] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:15.739231 12220 log.cc:1196] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/462182f031184f19b8f74a6d090545bd
I20260619 14:57:15.739528 12220 ts_tablet_manager.cc:1939] T 462182f031184f19b8f74a6d090545bd P 65c0dc1872f043dc91cd6718dfadff4a: Deleting consensus metadata
I20260619 14:57:15.740159 11626 catalog_manager.cc:4841] TS 65c0dc1872f043dc91cd6718dfadff4a (127.10.71.130:38337): tablet 462182f031184f19b8f74a6d090545bd (table test-table [id=4882946524024617b7b0904a79336c6a]) successfully deleted
I20260619 14:57:15.741328 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.729275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.10.71.129:35131)
0619 14:57:15.729372 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.729376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.730403 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 14:57:15.730567 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.730570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.730636 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.730716 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:15.731488 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.732358 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.733807 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.734145 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.734241 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.737703 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:15.737713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:15.737725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:15.737747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:15.739472 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.739908 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.739913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.739915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.740008 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.740317 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.740321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.740323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.740592 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:15.740749 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.741068 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.741184 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 14:57:15.742093 11839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42668
I20260619 14:57:15.752205 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:15.743681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.10.71.190:37899)
0619 14:57:15.743824 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:15.743828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:15.744235 (+ 407us) server_negotiation.cc:407] Connection header received
0619 14:57:15.744410 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:15.744413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:15.744470 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:15.744541 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:15.745285 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.746023 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.747044 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:15.747378 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:15.747469 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:15.748946 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:15.748950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:15.748952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:15.748974 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:15.750405 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.750842 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.750845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.750846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.750920 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:15.751206 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:15.751209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:15.751211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:15.751388 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:15.751496 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:15.751916 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 14:57:15.752019 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:57:15.753008 11639 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51062
I20260619 14:57:15.754277 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11714
I20260619 14:57:15.763504 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11854
I20260619 14:57:15.768828 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11994
I20260619 14:57:15.774765 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 11610
2026-06-19T14:57:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2701 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[12228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[12228](info): set up 2 sockets
Jun 19 14:57:15 dist-test-slave-3qdh krb5kdc[12228](info): commencing operation
krb5kdc: starting...
W20260619 14:57:16.739774 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:16 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:16.848925 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:45057
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:16.946518 12241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:16.946753 12241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:16.946790 12241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:16.950241 12241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:16.950285 12241 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:57:16.950302 12241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:16.950320 12241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:16.950374 12241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:16.950420 12241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:16.955261 12241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:45057
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:16.956058 12241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:16.956902 12241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:16.964375 12247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:16.964349 12246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:16.964495 12241 server_base.cc:1034] running on GCE node
W20260619 14:57:16.964339 12249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:16.965080 12241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:16.966125 12241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:16.967301 12241 hybrid_clock.cc:648] HybridClock initialized: now 1781881036967284 us; error 29 us; skew 500 ppm
Jun 19 14:57:16 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881036, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:16.969857 12241 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:16.970577 12241 webserver.cc:460] Webserver started at http://127.10.71.190:34229/ using document root <none> and password file <none>
I20260619 14:57:16.971006 12241 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:16.971055 12241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:16.971268 12241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:16.973284 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "25fe1e6161cb44c6bc991e60bedb8c6d"
format_stamp: "Formatted at 2026-06-19 14:57:16 on dist-test-slave-3qdh"
server_key: "5fac6930f02e1a4bf2eb26dc0d61dffe"
server_key_iv: "56f7aaf8ce98faff64b87a1689384e60"
server_key_version: "encryptionkey@0"
I20260619 14:57:16.973829 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "25fe1e6161cb44c6bc991e60bedb8c6d"
format_stamp: "Formatted at 2026-06-19 14:57:16 on dist-test-slave-3qdh"
server_key: "5fac6930f02e1a4bf2eb26dc0d61dffe"
server_key_iv: "56f7aaf8ce98faff64b87a1689384e60"
server_key_version: "encryptionkey@0"
I20260619 14:57:16.977058 12241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:16.979212 12256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:16.980356 12241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:16.980470 12241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "25fe1e6161cb44c6bc991e60bedb8c6d"
format_stamp: "Formatted at 2026-06-19 14:57:16 on dist-test-slave-3qdh"
server_key: "5fac6930f02e1a4bf2eb26dc0d61dffe"
server_key_iv: "56f7aaf8ce98faff64b87a1689384e60"
server_key_version: "encryptionkey@0"
I20260619 14:57:16.980566 12241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:16.994138 12241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:16.996564 12241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:16.996737 12241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:17.003670 12241 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:45057
I20260619 14:57:17.003715 12318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:45057 every 8 connection(s)
I20260619 14:57:17.004575 12241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:17.006847 12319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.011482 12319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Bootstrap starting.
I20260619 14:57:17.013545 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 12241
I20260619 14:57:17.013643 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:17.013805 12319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.013914 10526 external_mini_cluster.cc:1428] Setting key 7586431ada043061d8c10cf6274bf5d4
I20260619 14:57:17.014526 12319 log.cc:824] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:17.016446 12319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: No bootstrap required, opened a new log
I20260619 14:57:17.020097 12319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45057 } }
I20260619 14:57:17.020340 12319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.020395 12319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fe1e6161cb44c6bc991e60bedb8c6d, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.020781 12319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45057 } }
I20260619 14:57:17.020916 12319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:17.021005 12319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:17.021106 12319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:17.021872 12319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45057 } }
I20260619 14:57:17.022186 12319 leader_election.cc:302] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25fe1e6161cb44c6bc991e60bedb8c6d; no voters:
I20260619 14:57:17.022447 12319 leader_election.cc:288] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:17.022547 12324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:17.022807 12324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [term 1 LEADER]: Becoming Leader. State: Replica: 25fe1e6161cb44c6bc991e60bedb8c6d, State: Running, Role: LEADER
I20260619 14:57:17.023124 12324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45057 } }
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:17.023552 12319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:17.024322 12325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25fe1e6161cb44c6bc991e60bedb8c6d. Latest consensus state: current_term: 1 leader_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45057 } } }
I20260619 14:57:17.024441 12325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:17.024787 12333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:17.025869 12326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fe1e6161cb44c6bc991e60bedb8c6d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45057 } } }
I20260619 14:57:17.026026 12326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:17.027886 12333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:17.029167 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.014908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.10.71.190:45057)
0619 14:57:17.015284 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.015301 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.015338 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:57:17.016072 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.016091 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.016445 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.016769 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.017321 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.019477 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.020316 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.020965 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.021131 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.023864 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.023872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.023886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.023911 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.026187 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.026628 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.026653 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.026659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.026800 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.027134 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.027137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.027139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.027579 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.027752 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.028063 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.028346 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
I20260619 14:57:17.033640 12333 catalog_manager.cc:1347] Generated new cluster ID: cd59a2e0df6842b8a421b11fdc4e4892
I20260619 14:57:17.033699 12333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:17.055375 12333 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:17.056123 12333 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:17.060114 12333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Generated new TSK 0
I20260619 14:57:17.060679 12333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:17.086639 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:45057
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:17.211445 12345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:17.211653 12345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:17.211752 12345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:17.215482 12345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:17.215528 12345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:17.215561 12345 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:57:17.215610 12345 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:17.215667 12345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:17.220131 12345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:45057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:17.220974 12345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:17.221886 12345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:17.230362 12353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:17.230396 12350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:17.230396 12351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:17.230554 12345 server_base.cc:1034] running on GCE node
I20260619 14:57:17.230932 12345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:17.231460 12345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:17.232627 12345 hybrid_clock.cc:648] HybridClock initialized: now 1781881037232606 us; error 32 us; skew 500 ppm
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:17.235009 12345 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:17.235643 12345 webserver.cc:460] Webserver started at http://127.10.71.129:34935/ using document root <none> and password file <none>
I20260619 14:57:17.236039 12345 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:17.236089 12345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:17.236217 12345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:17.237871 12345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "016935878ce14659a476d2e05f851a91"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "ce0214d970b1827df897e2c58e57c7b0"
server_key_iv: "512a844098e7f2bc3891113d663f4331"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.238310 12345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "016935878ce14659a476d2e05f851a91"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "ce0214d970b1827df897e2c58e57c7b0"
server_key_iv: "512a844098e7f2bc3891113d663f4331"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.241402 12345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:17.243453 12360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:17.244525 12345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:17.244632 12345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "016935878ce14659a476d2e05f851a91"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "ce0214d970b1827df897e2c58e57c7b0"
server_key_iv: "512a844098e7f2bc3891113d663f4331"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.244719 12345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:17.257177 12345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:17.259382 12345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:17.259533 12345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:17.260701 12345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:17.260768 12345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:17.260812 12345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:17.260838 12345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:17.270808 12345 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:41903
I20260619 14:57:17.270850 12475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:41903 every 8 connection(s)
I20260619 14:57:17.271660 12345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:17.272123 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 12345
I20260619 14:57:17.272270 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:17.272598 10526 external_mini_cluster.cc:1428] Setting key e4283ef35a9ba857d2bdc8efa47ded9a
I20260619 14:57:17.274812 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.261233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:48425 (local address 127.10.71.190:45057)
0619 14:57:17.261338 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.261348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.262665 (+ 1317us) server_negotiation.cc:407] Connection header received
0619 14:57:17.263787 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.263790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.263847 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.263927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.265320 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.266192 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.268015 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.268234 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.268363 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.271251 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.271255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.271258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.271287 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.272929 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.273454 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.273457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.273458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.273532 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.273847 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.273850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.273852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.274058 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.274131 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.274626 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.274711 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:57:17.275732 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.262117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.10.71.129:48425)
0619 14:57:17.262522 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.262557 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.263557 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.264108 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.264117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.264555 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.265163 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.265182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.266384 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.266391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.267887 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.267897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.268407 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.268413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.268618 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.269230 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.269250 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.271086 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.273052 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.273058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.273069 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.273366 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.273614 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.273615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.273617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.273763 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.274159 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.274164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.274411 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.274601 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.274889 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260619 14:57:17.277097 12476 heartbeater.cc:347] Connected to a master server at 127.10.71.190:45057
I20260619 14:57:17.277230 12476 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:17.277694 12476 heartbeater.cc:511] Master 127.10.71.190:45057 requested a full tablet report, sending...
I20260619 14:57:17.279222 12272 ts_manager.cc:194] Registered new tserver with Master: 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903)
I20260619 14:57:17.280669 12272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:48425
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:17.287458 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.278857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:53855 (local address 127.10.71.190:45057)
0619 14:57:17.278953 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.278957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.280560 (+ 1603us) server_negotiation.cc:407] Connection header received
0619 14:57:17.280683 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.280685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.280725 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.280800 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.281247 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.282056 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.283094 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.283277 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.283341 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.284353 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.284356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.284359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.284383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.285968 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.286357 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.286362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.286364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.286441 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.286710 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.286715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.286716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.286867 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.286962 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.287215 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.287316 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:57:17.287463 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.278815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.10.71.129:53855)
0619 14:57:17.280484 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.280494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.280593 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.280901 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.280903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.281039 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.281115 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.281119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.282204 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.282208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.282983 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.282991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.283386 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.283389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.283459 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.283836 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.283846 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.284212 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.286079 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.286082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.286084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.286263 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.286525 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.286529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.286531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.286614 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.286954 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.286956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.287035 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.287165 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.287290 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1609,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:17.298702 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:45057
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:17.396132 12485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:17.396361 12485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:17.396462 12485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:17.400197 12485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:17.400244 12485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:17.400285 12485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:57:17.400368 12485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:17.400414 12485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:17.405217 12485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:45057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:17.405941 12485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:17.406702 12485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:17.414881 12493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:17.414891 12490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:17.415014 12485 server_base.cc:1034] running on GCE node
W20260619 14:57:17.414855 12491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:17.415428 12485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:17.415968 12485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:17.417130 12485 hybrid_clock.cc:648] HybridClock initialized: now 1781881037417108 us; error 37 us; skew 500 ppm
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:17.419423 12485 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:17.420104 12485 webserver.cc:460] Webserver started at http://127.10.71.130:44279/ using document root <none> and password file <none>
I20260619 14:57:17.420573 12485 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:17.420652 12485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:17.420859 12485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:17.422547 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "63207c115b1c45b1a6e8c0f3c6f249f1"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "6032dad7ae7b055d7fabcb2e340e6a04"
server_key_iv: "b770e09eb392fb00e8e05ed82cc21daa"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.423128 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "63207c115b1c45b1a6e8c0f3c6f249f1"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "6032dad7ae7b055d7fabcb2e340e6a04"
server_key_iv: "b770e09eb392fb00e8e05ed82cc21daa"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.426299 12485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:57:17.428273 12500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:17.429334 12485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:17.429469 12485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "63207c115b1c45b1a6e8c0f3c6f249f1"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "6032dad7ae7b055d7fabcb2e340e6a04"
server_key_iv: "b770e09eb392fb00e8e05ed82cc21daa"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.429579 12485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:17.445066 12485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:17.447378 12485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:17.447533 12485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:17.448830 12485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:17.448892 12485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:17.448941 12485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:17.448976 12485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:17.458685 12485 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:43397
I20260619 14:57:17.458734 12615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:43397 every 8 connection(s)
I20260619 14:57:17.459358 12485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:17.460734 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.449512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:52845 (local address 127.10.71.190:45057)
0619 14:57:17.449644 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.449647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.450293 (+ 646us) server_negotiation.cc:407] Connection header received
0619 14:57:17.451068 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.451072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.451131 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.451223 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.452653 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.453328 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.454516 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.454710 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.454807 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.457374 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.457377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.457379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.457402 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.458915 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.459360 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.459364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.459366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.459445 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.459801 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.459804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.459805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.459978 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.460057 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.460491 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.460589 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:17.461308 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.449796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.10.71.130:52845)
0619 14:57:17.450172 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.450204 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.450877 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.451367 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.451375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.451823 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.452494 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.452513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.453445 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.453449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.454403 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.454412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.454836 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.454839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.455011 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.455514 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.455530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.457235 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.459022 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.459027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.459038 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.459265 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.459530 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.459532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.459534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.459701 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.460062 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.460069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.460251 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.460454 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.460699 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260619 14:57:17.462354 12616 heartbeater.cc:347] Connected to a master server at 127.10.71.190:45057
I20260619 14:57:17.462497 12616 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:17.462946 12616 heartbeater.cc:511] Master 127.10.71.190:45057 requested a full tablet report, sending...
I20260619 14:57:17.463330 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 12485
I20260619 14:57:17.463428 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:17.463680 10526 external_mini_cluster.cc:1428] Setting key 4a18f0fd84512f775581e1041e24402e
I20260619 14:57:17.464432 12273 ts_manager.cc:194] Registered new tserver with Master: 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397)
I20260619 14:57:17.465015 12273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:52845
I20260619 14:57:17.470645 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.465210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:60373 (local address 127.10.71.190:45057)
0619 14:57:17.465311 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.465316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.465329 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:17.465355 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.465357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.465391 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.465469 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.465855 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.466261 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.466941 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.467135 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.467227 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.468072 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.468075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.468077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.468096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.469277 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.469630 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.469634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.469636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.469701 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.469939 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.469943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.469944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.470116 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.470213 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.470436 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.470526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:17.470664 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.464378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.10.71.130:60373)
0619 14:57:17.464617 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.464630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.464703 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.465484 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.465486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.465645 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.465758 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.465763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.466367 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.466369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.466839 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.466846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.467230 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.467233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.467315 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.467713 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.467722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.467960 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.469375 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.469379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.469381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.469544 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.469780 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.469784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.469785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.469865 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.470205 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.470208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.470281 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.470374 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.470505 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":12,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:17.486960 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:45057
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:17.582070 12625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:17.582267 12625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:17.582310 12625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:17.586081 12625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:17.586125 12625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:17.586159 12625 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:57:17.586172 12625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:17.586199 12625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:17.590466 12625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:45057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:17.591202 12625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:17.591995 12625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:17.600162 12630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:17.600163 12631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:17.600188 12633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:17.600350 12625 server_base.cc:1034] running on GCE node
I20260619 14:57:17.600788 12625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:17.601289 12625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:17.602439 12625 hybrid_clock.cc:648] HybridClock initialized: now 1781881037602424 us; error 27 us; skew 500 ppm
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:17.604863 12625 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:17.605623 12625 webserver.cc:460] Webserver started at http://127.10.71.131:43657/ using document root <none> and password file <none>
I20260619 14:57:17.606097 12625 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:17.606151 12625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:17.606361 12625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:17.608040 12625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "5cafa50e9f2f4202867968dc0954b659"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "cf75901ed98440992d9a48ed364b5daa"
server_key_iv: "33006068ccba6bf9aa7105f5a551dbab"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.608546 12625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "5cafa50e9f2f4202867968dc0954b659"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "cf75901ed98440992d9a48ed364b5daa"
server_key_iv: "33006068ccba6bf9aa7105f5a551dbab"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.611555 12625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:57:17.613747 12640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:17.614763 12625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:17.614876 12625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "5cafa50e9f2f4202867968dc0954b659"
format_stamp: "Formatted at 2026-06-19 14:57:17 on dist-test-slave-3qdh"
server_key: "cf75901ed98440992d9a48ed364b5daa"
server_key_iv: "33006068ccba6bf9aa7105f5a551dbab"
server_key_version: "encryptionkey@0"
I20260619 14:57:17.614965 12625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:17.635103 12625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:17.638250 12625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:17.638437 12625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:17.639760 12625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:17.639829 12625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:17.639894 12625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:17.639936 12625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:17.650945 12625 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:36153
I20260619 14:57:17.651227 12755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:36153 every 8 connection(s)
I20260619 14:57:17.651854 12625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:17.653915 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.640525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:39429 (local address 127.10.71.190:45057)
0619 14:57:17.640673 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.640678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.641567 (+ 889us) server_negotiation.cc:407] Connection header received
0619 14:57:17.642531 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.642535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.642602 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.642663 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.644123 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.644847 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.646034 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.646363 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.646477 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.650044 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.650048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.650052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.650087 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.651697 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.652285 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.652292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.652310 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.652419 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.652824 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.652828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.652830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.653024 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.653145 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.653604 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.653735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:57:17.654596 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.640972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.10.71.131:39429)
0619 14:57:17.641377 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.641444 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.642293 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.643110 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.643116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.643473 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.643972 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.643994 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.644967 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.644971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.645906 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.645915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.646522 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.646526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.646678 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.647252 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.647274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.649862 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.651826 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.651832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.651845 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.652173 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.652559 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.652562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.652564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.652709 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.653151 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.653160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.653370 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.653567 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.653852 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260619 14:57:17.655779 12756 heartbeater.cc:347] Connected to a master server at 127.10.71.190:45057
I20260619 14:57:17.655954 12756 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:17.656471 12756 heartbeater.cc:511] Master 127.10.71.190:45057 requested a full tablet report, sending...
I20260619 14:57:17.657486 12272 ts_manager.cc:194] Registered new tserver with Master: 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:17.658187 12272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:39429
I20260619 14:57:17.661330 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 12625
I20260619 14:57:17.661414 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:17.661681 10526 external_mini_cluster.cc:1428] Setting key e55fba34f3ae6ab307b062c71c617780
I20260619 14:57:17.663630 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:57:17.665058 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.657874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:54253 (local address 127.10.71.190:45057)
0619 14:57:17.658003 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.658007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.658146 (+ 139us) server_negotiation.cc:407] Connection header received
0619 14:57:17.658246 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.658248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.658322 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.658423 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.658924 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.659677 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.660519 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.660710 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.660786 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.661914 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.661918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.661920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.661942 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.663398 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.663875 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.663878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.663880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.663954 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.664201 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.664205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.664207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.664434 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.664529 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.664779 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.664878 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:57:17.665071 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.657808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.10.71.131:54253)
0619 14:57:17.658063 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.658076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.658166 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.658426 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.658430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.658672 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.658794 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.658800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.659805 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.659807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.660425 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.660434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.660798 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.660800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.660861 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.661326 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.661337 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.661703 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.663510 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.663512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.663514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.663773 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.664036 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.664040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.664042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.664120 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.664530 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.664533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.664627 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.664729 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.664854 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:17.686872 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.679631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.10.71.190:45057)
0619 14:57:17.679741 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.679743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.679753 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:57:17.679780 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.679782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.679810 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.679868 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.680501 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.680952 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.681771 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.681966 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.682042 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.684236 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.684239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.684241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.684261 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.685618 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.685961 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.685963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.685964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.686028 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.686257 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.686259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.686260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.686397 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.686510 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.686694 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.686773 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:17.689429 12272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:17.692226 12272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:17.705713 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.699657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36153 (local address 127.0.0.1:41578)
0619 14:57:17.699950 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.699993 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.700114 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.701193 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.701197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.701216 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:17.701300 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.701304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.702731 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.702734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.704446 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.704454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.705204 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.705207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.705313 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.705352 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.705485 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.705590 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":187,"spinlock_wait_cycles":2304,"thread_start_us":94,"threads_started":1}
I20260619 14:57:17.705744 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.700284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.10.71.131:36153)
0619 14:57:17.700714 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.700731 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.700753 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:57:17.700826 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.700833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.701031 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.701189 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:17.701451 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.702615 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.704553 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.705079 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.705197 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.705354 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.705510 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.705591 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":110,"threads_started":1}
I20260619 14:57:17.706136 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.699740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:41903 (local address 127.0.0.1:43554)
0619 14:57:17.699974 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.700005 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.700113 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.701786 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.701788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.701804 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:17.701895 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.701901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.702923 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.702927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.704492 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.704503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.705608 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.705611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.705697 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.705718 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.705919 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.705988 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":117,"spinlock_wait_cycles":11264,"thread_start_us":87,"threads_started":1}
I20260619 14:57:17.706444 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.700551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.0.0.1:45974)
0619 14:57:17.701415 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.701445 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.701535 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.701959 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.701965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.701983 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:17.702055 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.702061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.703893 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.703896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.705084 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.705102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.706099 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.706103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.706181 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.706210 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.706272 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.706331 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":33,"thread_start_us":99,"threads_started":1}
I20260619 14:57:17.706892 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.700454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.10.71.129:41903)
0619 14:57:17.701330 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.701340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.701362 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:57:17.701460 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.701469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.701659 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.701811 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:17.702017 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.702795 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.704650 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.705478 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.706372 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.706611 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.706674 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.706758 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":70,"threads_started":1}
I20260619 14:57:17.707109 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.700745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.10.71.130:43397)
0619 14:57:17.701094 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.701106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.701537 (+ 431us) server_negotiation.cc:407] Connection header received
0619 14:57:17.701624 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.701629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.701809 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.701945 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:17.702158 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.703775 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.705220 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.705987 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.706085 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.706814 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.706855 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.706953 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":79,"threads_started":1}
I20260619 14:57:17.708772 12690 tablet_service.cc:1467] Processing CreateTablet for tablet 767c6625c3934aa5b00384934bd848d7 (DEFAULT_TABLE table=test-table [id=e5fd98aef84b4433bd63d3cf5fd403c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:17.709637 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767c6625c3934aa5b00384934bd848d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.709478 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 767c6625c3934aa5b00384934bd848d7 (DEFAULT_TABLE table=test-table [id=e5fd98aef84b4433bd63d3cf5fd403c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:17.709959 12550 tablet_service.cc:1467] Processing CreateTablet for tablet 767c6625c3934aa5b00384934bd848d7 (DEFAULT_TABLE table=test-table [id=e5fd98aef84b4433bd63d3cf5fd403c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:17.710471 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767c6625c3934aa5b00384934bd848d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.710804 12550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767c6625c3934aa5b00384934bd848d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.715831 12779 tablet_bootstrap.cc:492] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Bootstrap starting.
I20260619 14:57:17.717696 12781 tablet_bootstrap.cc:492] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Bootstrap starting.
I20260619 14:57:17.718005 12779 tablet_bootstrap.cc:654] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.718896 12779 log.cc:824] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:17.719492 12780 tablet_bootstrap.cc:492] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Bootstrap starting.
I20260619 14:57:17.719664 12781 tablet_bootstrap.cc:654] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.720412 12781 log.cc:824] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:17.720685 12779 tablet_bootstrap.cc:492] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: No bootstrap required, opened a new log
I20260619 14:57:17.720857 12779 ts_tablet_manager.cc:1397] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:17.721415 12780 tablet_bootstrap.cc:654] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.722043 12780 log.cc:824] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:17.722205 12781 tablet_bootstrap.cc:492] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: No bootstrap required, opened a new log
I20260619 14:57:17.722368 12781 ts_tablet_manager.cc:1397] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 14:57:17.723644 12780 tablet_bootstrap.cc:492] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: No bootstrap required, opened a new log
I20260619 14:57:17.723941 12780 ts_tablet_manager.cc:1397] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:17.724831 12779 raft_consensus.cc:348] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.725087 12779 raft_consensus.cc:374] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.725142 12779 raft_consensus.cc:729] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cafa50e9f2f4202867968dc0954b659, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.725543 12779 consensus_queue.cc:260] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.726172 12756 heartbeater.cc:503] Master 127.10.71.190:45057 was elected leader, sending a full tablet report...
I20260619 14:57:17.726068 12781 raft_consensus.cc:348] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.726291 12781 raft_consensus.cc:374] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.726344 12781 raft_consensus.cc:729] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.726529 12779 ts_tablet_manager.cc:1428] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:57:17.726781 12781 consensus_queue.cc:260] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.727664 12781 ts_tablet_manager.cc:1428] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 14:57:17.727638 12780 raft_consensus.cc:348] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.727931 12780 raft_consensus.cc:374] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.727982 12780 raft_consensus.cc:729] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016935878ce14659a476d2e05f851a91, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.727986 12616 heartbeater.cc:503] Master 127.10.71.190:45057 was elected leader, sending a full tablet report...
I20260619 14:57:17.728376 12780 consensus_queue.cc:260] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.728951 12476 heartbeater.cc:503] Master 127.10.71.190:45057 was elected leader, sending a full tablet report...
I20260619 14:57:17.729285 12780 ts_tablet_manager.cc:1428] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:17.758705 12787 raft_consensus.cc:482] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:17.758877 12787 raft_consensus.cc:504] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.759805 12787 leader_election.cc:288] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153), 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397)
I20260619 14:57:17.764438 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.760488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:48523 (local address 127.10.71.130:43397)
0619 14:57:17.760621 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.760627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.760639 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:17.760803 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.760806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.760867 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.760937 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:17.761155 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.761882 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.763264 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.763832 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.763936 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.763978 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.764257 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.764332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:57:17.764437 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.760099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:58923 (local address 127.10.71.131:36153)
0619 14:57:17.760216 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.760220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.760232 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:17.760431 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.760435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.760483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.760566 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:17.760896 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.761710 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.763227 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.763832 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.763936 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.763978 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.764258 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.764332 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:17.764463 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.760035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36153 (local address 127.10.71.129:58923)
0619 14:57:17.760169 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.760184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.760262 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.760603 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.760606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.760642 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:17.760786 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.760792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.761856 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.761859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.763078 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.763098 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.763933 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.763936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.764111 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.764159 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.764226 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.764329 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":100,"spinlock_wait_cycles":13056}
I20260619 14:57:17.764474 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.760321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.10.71.129:48523)
0619 14:57:17.760547 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.760561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.760656 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.760953 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.760955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.760969 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:17.761058 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.761062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.761972 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.761975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.763159 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.763169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.763933 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.763936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.764092 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.764161 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.764226 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.764329 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":57,"spinlock_wait_cycles":1280,"thread_start_us":79,"threads_started":1}
I20260619 14:57:17.765030 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767c6625c3934aa5b00384934bd848d7" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659" is_pre_election: true
I20260619 14:57:17.765266 12710 raft_consensus.cc:2454] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 0.
I20260619 14:57:17.765255 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767c6625c3934aa5b00384934bd848d7" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" is_pre_election: true
I20260619 14:57:17.765479 12570 raft_consensus.cc:2454] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 0.
I20260619 14:57:17.765726 12361 leader_election.cc:302] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 5cafa50e9f2f4202867968dc0954b659; no voters:
I20260619 14:57:17.765936 12787 raft_consensus.cc:2781] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:17.766013 12787 raft_consensus.cc:482] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:17.766042 12787 raft_consensus.cc:3037] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:17.767328 12787 raft_consensus.cc:504] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.767618 12787 leader_election.cc:288] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 election: Requested vote from peers 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153), 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397)
I20260619 14:57:17.767974 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767c6625c3934aa5b00384934bd848d7" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1"
I20260619 14:57:17.767971 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767c6625c3934aa5b00384934bd848d7" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659"
I20260619 14:57:17.768077 12570 raft_consensus.cc:3037] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:17.768077 12710 raft_consensus.cc:3037] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:17.769361 12710 raft_consensus.cc:2454] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 1.
I20260619 14:57:17.769361 12570 raft_consensus.cc:2454] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 1.
I20260619 14:57:17.769620 12361 leader_election.cc:302] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 5cafa50e9f2f4202867968dc0954b659; no voters:
I20260619 14:57:17.769761 12787 raft_consensus.cc:2781] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:17.769968 12787 raft_consensus.cc:686] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 1 LEADER]: Becoming Leader. State: Replica: 016935878ce14659a476d2e05f851a91, State: Running, Role: LEADER
I20260619 14:57:17.770254 12787 consensus_queue.cc:237] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } }
I20260619 14:57:17.772960 12272 catalog_manager.cc:5495] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 016935878ce14659a476d2e05f851a91 (127.10.71.129). New cstate: current_term: 1 leader_uuid: "016935878ce14659a476d2e05f851a91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } health_report { overall_health: UNKNOWN } } }
W20260619 14:57:17.773561 12477 tablet.cc:2369] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:57:17 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881036, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:17.817184 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.809591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45057 (local address 127.0.0.1:49358)
0619 14:57:17.809920 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:17.809931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:17.810039 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:17.810298 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:17.810301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:17.810677 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:17.810756 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.810761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.811459 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.811465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.812031 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:17.812038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.812517 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.812521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:17.812597 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.813204 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:17.813214 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:17.814412 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:17.815691 (+ 1279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.815698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.815702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.815897 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.816138 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:17.816140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:17.816142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:17.816220 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:17.816691 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:17.816695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:17.816808 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 14:57:17.816906 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:57:17.817022 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260619 14:57:17.817205 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:17.809671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.10.71.190:45057)
0619 14:57:17.809848 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:17.809854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:17.810002 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:57:17.810162 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:17.810167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:17.810207 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:17.810284 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:17.810874 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.811355 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.812152 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:17.812414 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:17.812517 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:17.814525 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:17.814528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:17.814529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:17.814548 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:17.815606 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.815973 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.815976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.815977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.816039 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:17.816435 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:17.816437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:17.816438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:17.816580 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:17.816649 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:17.816954 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 14:57:17.817084 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":52}
I20260619 14:57:17.820703 12272 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:49358:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:57:17.821275 12272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:17.825660 12410 tablet_service.cc:1467] Processing CreateTablet for tablet f0c8d03f7ccc45eb896b92c431b2bc77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=916e3f2cc50b4e9f82e424600a67012b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:17.825924 12690 tablet_service.cc:1467] Processing CreateTablet for tablet f0c8d03f7ccc45eb896b92c431b2bc77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=916e3f2cc50b4e9f82e424600a67012b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:17.825924 12550 tablet_service.cc:1467] Processing CreateTablet for tablet f0c8d03f7ccc45eb896b92c431b2bc77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=916e3f2cc50b4e9f82e424600a67012b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:17.826359 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c8d03f7ccc45eb896b92c431b2bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.826519 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c8d03f7ccc45eb896b92c431b2bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.826519 12550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c8d03f7ccc45eb896b92c431b2bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:17.828954 12780 tablet_bootstrap.cc:492] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: Bootstrap starting.
I20260619 14:57:17.829185 12781 tablet_bootstrap.cc:492] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: Bootstrap starting.
I20260619 14:57:17.829396 12779 tablet_bootstrap.cc:492] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: Bootstrap starting.
I20260619 14:57:17.829828 12780 tablet_bootstrap.cc:654] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.830060 12781 tablet_bootstrap.cc:654] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.830114 12779 tablet_bootstrap.cc:654] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:17.831002 12780 tablet_bootstrap.cc:492] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: No bootstrap required, opened a new log
I20260619 14:57:17.831079 12780 ts_tablet_manager.cc:1397] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:17.831133 12779 tablet_bootstrap.cc:492] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: No bootstrap required, opened a new log
I20260619 14:57:17.831203 12779 ts_tablet_manager.cc:1397] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:17.831262 12781 tablet_bootstrap.cc:492] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: No bootstrap required, opened a new log
I20260619 14:57:17.831341 12781 ts_tablet_manager.cc:1397] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:57:17.831488 12780 raft_consensus.cc:348] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:17.831606 12780 raft_consensus.cc:374] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.831550 12779 raft_consensus.cc:348] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:17.831642 12780 raft_consensus.cc:729] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016935878ce14659a476d2e05f851a91, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.831650 12779 raft_consensus.cc:374] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.831683 12779 raft_consensus.cc:729] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cafa50e9f2f4202867968dc0954b659, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.831809 12780 consensus_queue.cc:260] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:17.831842 12779 consensus_queue.cc:260] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:17.831827 12781 raft_consensus.cc:348] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:17.831929 12781 raft_consensus.cc:374] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:17.831965 12781 raft_consensus.cc:729] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Initialized, Role: FOLLOWER
I20260619 14:57:17.832072 12779 ts_tablet_manager.cc:1428] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:17.832072 12780 ts_tablet_manager.cc:1428] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:17.832101 12781 consensus_queue.cc:260] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:17.832319 12785 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:17.832394 12781 ts_tablet_manager.cc:1428] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:17.832451 12785 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:17.832649 12787 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:17.832765 12787 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:17.832962 12786 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:17.833107 12786 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 14:57:17.903152 12757 tablet.cc:2369] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:57:17.960444 12617 tablet.cc:2369] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:18.051261 12786 raft_consensus.cc:482] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:18.051389 12786 raft_consensus.cc:504] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.052618 12786 leader_election.cc:288] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903), 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:18.056496 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.052982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:55099 (local address 127.10.71.129:41903)
0619 14:57:18.053134 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.053139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.053303 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:57:18.053427 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.053430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.053489 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.053557 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:18.053866 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.054368 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.055538 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.055943 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.056060 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.056119 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.056290 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.056380 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:57:18.056568 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.052869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:41903 (local address 127.10.71.130:55099)
0619 14:57:18.053219 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:18.053230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:18.053323 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:18.053597 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:18.053602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:18.053624 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:18.053755 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.053760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.054474 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.054480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.055428 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.055440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.056066 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.056070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.056170 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.056191 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:57:18.056269 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:57:18.056381 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":57,"thread_start_us":94,"threads_started":1}
I20260619 14:57:18.056977 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.052999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:42825 (local address 127.10.71.131:36153)
0619 14:57:18.053142 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.053146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.053401 (+ 255us) server_negotiation.cc:407] Connection header received
0619 14:57:18.053509 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.053512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.053576 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.053648 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:18.053905 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.054393 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.055462 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.056026 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.056147 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.056194 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.056792 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.056858 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:57:18.057327 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.052869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36153 (local address 127.10.71.130:42825)
0619 14:57:18.053299 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:18.053313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:18.053416 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:18.053673 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:18.053676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:18.053696 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:18.053776 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.053780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.054521 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.054523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.055353 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.055361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.056534 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.056539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.056640 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.056672 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:57:18.057140 (+ 468us) client_negotiation.cc:240] Negotiation successful
0619 14:57:18.057208 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":52,"thread_start_us":89,"threads_started":1}
I20260619 14:57:18.057319 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8d03f7ccc45eb896b92c431b2bc77" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016935878ce14659a476d2e05f851a91" is_pre_election: true
I20260619 14:57:18.057587 12430 raft_consensus.cc:2454] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 0.
I20260619 14:57:18.057785 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8d03f7ccc45eb896b92c431b2bc77" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659" is_pre_election: true
I20260619 14:57:18.057950 12710 raft_consensus.cc:2454] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 0.
I20260619 14:57:18.058048 12503 leader_election.cc:302] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:18.058391 12786 raft_consensus.cc:2781] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:18.058449 12786 raft_consensus.cc:482] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:18.058475 12786 raft_consensus.cc:3037] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:18.059453 12786 raft_consensus.cc:504] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.059796 12786 leader_election.cc:288] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 election: Requested vote from peers 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903), 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:18.060109 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8d03f7ccc45eb896b92c431b2bc77" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659"
I20260619 14:57:18.060111 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8d03f7ccc45eb896b92c431b2bc77" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016935878ce14659a476d2e05f851a91"
I20260619 14:57:18.060204 12710 raft_consensus.cc:3037] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:18.060231 12430 raft_consensus.cc:3037] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:18.061048 12710 raft_consensus.cc:2454] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 1.
I20260619 14:57:18.061229 12430 raft_consensus.cc:2454] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 1.
I20260619 14:57:18.061340 12501 leader_election.cc:302] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cafa50e9f2f4202867968dc0954b659, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:18.061519 12786 raft_consensus.cc:2781] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:18.061681 12786 raft_consensus.cc:686] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 LEADER]: Becoming Leader. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Running, Role: LEADER
I20260619 14:57:18.062009 12786 consensus_queue.cc:237] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.063984 12802 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: TxnStatusTablet state changed. Reason: New leader 63207c115b1c45b1a6e8c0f3c6f249f1. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.064090 12802 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:18.064393 12804 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:57:18.064589 12804 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:57:18.065196 12272 catalog_manager.cc:5495] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130). New cstate: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:18.122632 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.118748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.0.0.1:45984)
0619 14:57:18.119080 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:18.119093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:18.119218 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:18.119505 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:18.119507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:18.119513 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:18.119606 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.119610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.120521 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.120523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.121083 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.121092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.121578 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.121581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.121643 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.121680 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.122381 (+ 701us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.122391 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:57:18.122443 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:57:18.122514 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260619 14:57:18.122648 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.118880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.10.71.130:43397)
0619 14:57:18.119018 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.119022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.119186 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:57:18.119331 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.119333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.119391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.119485 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:18.119704 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.120419 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.121187 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.121476 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.121571 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.121820 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.122289 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.122398 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.122464 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.122513 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:57:18.127825 12710 raft_consensus.cc:1261] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Refusing update from remote peer 63207c115b1c45b1a6e8c0f3c6f249f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:18.127857 12430 raft_consensus.cc:1261] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Refusing update from remote peer 63207c115b1c45b1a6e8c0f3c6f249f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:18.128465 12802 consensus_queue.cc:1035] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:18.128656 12786 consensus_queue.cc:1035] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:18.131295 12787 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: TxnStatusTablet state changed. Reason: New leader 63207c115b1c45b1a6e8c0f3c6f249f1. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.131398 12787 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:18.131735 12785 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: TxnStatusTablet state changed. Reason: New leader 63207c115b1c45b1a6e8c0f3c6f249f1. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.131908 12785 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:18.134667 12802 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.134758 12802 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:18.134805 12785 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.134889 12785 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 5cafa50e9f2f4202867968dc0954b659: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:18.134877 12802 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.134941 12802 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 63207c115b1c45b1a6e8c0f3c6f249f1: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:18.135075 12787 tablet_replica.cc:439] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } } }
I20260619 14:57:18.135150 12787 tablet_replica.cc:442] T f0c8d03f7ccc45eb896b92c431b2bc77 P 016935878ce14659a476d2e05f851a91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:18.136809 12808 mvcc.cc:205] Tried to move back new op lower bound from 7298584732165517312 to 7298584731904581632. Current Snapshot: MvccSnapshot[applied={T|T < 7298584732165517312}]
I20260619 14:57:18.138240 12809 mvcc.cc:205] Tried to move back new op lower bound from 7298584732165517312 to 7298584731904581632. Current Snapshot: MvccSnapshot[applied={T|T < 7298584732165517312}]
I20260619 14:57:18.145216 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.141871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.10.71.129:41903)
0619 14:57:18.141983 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.141987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.142000 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:18.142116 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.142118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.142164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.142287 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:18.142547 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.143068 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.143890 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.144093 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.144195 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.144493 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.144815 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.144935 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.145029 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.145087 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:18.155601 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.151863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:52473 (local address 127.10.71.130:43397)
0619 14:57:18.151996 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.152000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.152218 (+ 218us) server_negotiation.cc:407] Connection header received
0619 14:57:18.152319 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.152321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.152367 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.152445 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:18.152681 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.153348 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.154338 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.154670 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.154773 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.155159 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.155240 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.155311 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.155445 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.155490 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:57:18.155678 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.151712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.10.71.129:52473)
0619 14:57:18.152129 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:18.152142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:18.152227 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:18.152464 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:18.152466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:18.152472 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:18.152575 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.152580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.153448 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.153450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.154223 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.154232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.154782 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.154785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.154855 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.154893 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.155344 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.155352 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:57:18.155416 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:57:18.155479 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":133,"threads_started":1}
I20260619 14:57:18.161347 12710 raft_consensus.cc:1261] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Refusing update from remote peer 016935878ce14659a476d2e05f851a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:18.161443 12570 raft_consensus.cc:1261] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Refusing update from remote peer 016935878ce14659a476d2e05f851a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:18.161798 12789 consensus_queue.cc:1035] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:18.161940 12787 consensus_queue.cc:1035] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:18.168763 12807 mvcc.cc:205] Tried to move back new op lower bound from 7298584732304908288 to 7298584730709241856. Current Snapshot: MvccSnapshot[applied={T|T < 7298584732304908288}]
I20260619 14:57:18.185096 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.180841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:34943 (local address 127.10.71.129:41903)
0619 14:57:18.180970 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.180978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.181232 (+ 254us) server_negotiation.cc:407] Connection header received
0619 14:57:18.181392 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.181395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.181441 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.181533 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:18.181835 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.182547 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.183777 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.184042 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.184226 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.184569 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.184663 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.184780 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.184928 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.184979 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:57:18.185501 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.180762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:41903 (local address 127.10.71.130:34943)
0619 14:57:18.181143 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:18.181155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:18.181281 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:18.181551 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:18.181554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:18.181562 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:18.181689 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.181714 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.182697 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.182700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.183674 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.183685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.184248 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.184252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.184375 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.184426 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.184799 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.184807 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:57:18.185272 (+ 465us) client_negotiation.cc:240] Negotiation successful
0619 14:57:18.185348 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":107,"threads_started":1}
I20260619 14:57:18.211190 12273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:18.211382 12273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:18.213907 12273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Sending DeleteTablet for 3 replicas of tablet 767c6625c3934aa5b00384934bd848d7
I20260619 14:57:18.214542 12690 tablet_service.cc:1514] Processing DeleteTablet for tablet 767c6625c3934aa5b00384934bd848d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:18 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260619 14:57:18.214542 12410 tablet_service.cc:1514] Processing DeleteTablet for tablet 767c6625c3934aa5b00384934bd848d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:18 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:43554
I20260619 14:57:18.214911 12827 tablet_replica.cc:330] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: stopping tablet replica
I20260619 14:57:18.215230 12827 raft_consensus.cc:2229] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:18.215457 12827 raft_consensus.cc:2258] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:18.215469 12273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:18.215806 12273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:18.217027 12827 ts_tablet_manager.cc:1905] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:18.218297 12550 tablet_service.cc:1514] Processing DeleteTablet for tablet 767c6625c3934aa5b00384934bd848d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:18 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260619 14:57:18.220054 12549 tablet_service.cc:1467] Processing CreateTablet for tablet f5e17d3ea29843cc8a4371f3a0112211 (DEFAULT_TABLE table=test-table [id=2939844a09064918907e54d61d91c39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:18.220340 12549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e17d3ea29843cc8a4371f3a0112211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:18.221503 12827 ts_tablet_manager.cc:1918] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:18.221669 12827 log.cc:1196] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/767c6625c3934aa5b00384934bd848d7
I20260619 14:57:18.222115 12827 ts_tablet_manager.cc:1939] T 767c6625c3934aa5b00384934bd848d7 P 5cafa50e9f2f4202867968dc0954b659: Deleting consensus metadata
I20260619 14:57:18.222429 12781 tablet_bootstrap.cc:492] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Bootstrap starting.
I20260619 14:57:18.222678 12690 tablet_service.cc:1467] Processing CreateTablet for tablet f5e17d3ea29843cc8a4371f3a0112211 (DEFAULT_TABLE table=test-table [id=2939844a09064918907e54d61d91c39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:18.222941 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e17d3ea29843cc8a4371f3a0112211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:18.223177 12410 tablet_service.cc:1467] Processing CreateTablet for tablet f5e17d3ea29843cc8a4371f3a0112211 (DEFAULT_TABLE table=test-table [id=2939844a09064918907e54d61d91c39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:18.223358 12781 tablet_bootstrap.cc:654] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:18.223397 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e17d3ea29843cc8a4371f3a0112211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:18.223665 12257 catalog_manager.cc:4841] TS 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153): tablet 767c6625c3934aa5b00384934bd848d7 (table test-table [id=e5fd98aef84b4433bd63d3cf5fd403c5]) successfully deleted
I20260619 14:57:18.224915 12779 tablet_bootstrap.cc:492] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Bootstrap starting.
I20260619 14:57:18.225689 12780 tablet_bootstrap.cc:492] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Bootstrap starting.
I20260619 14:57:18.225752 12779 tablet_bootstrap.cc:654] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:18.226465 12780 tablet_bootstrap.cc:654] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:18.226867 12781 tablet_bootstrap.cc:492] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: No bootstrap required, opened a new log
I20260619 14:57:18.226946 12781 ts_tablet_manager.cc:1397] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 14:57:18.227037 12779 tablet_bootstrap.cc:492] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: No bootstrap required, opened a new log
I20260619 14:57:18.227118 12779 ts_tablet_manager.cc:1397] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:18.227324 12781 raft_consensus.cc:348] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.227422 12781 raft_consensus.cc:374] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:18.227453 12781 raft_consensus.cc:729] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Initialized, Role: FOLLOWER
I20260619 14:57:18.227497 12779 raft_consensus.cc:348] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.227604 12779 raft_consensus.cc:374] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:18.227553 12781 consensus_queue.cc:260] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.227636 12779 raft_consensus.cc:729] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cafa50e9f2f4202867968dc0954b659, State: Initialized, Role: FOLLOWER
I20260619 14:57:18.227754 12779 consensus_queue.cc:260] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.227859 12781 ts_tablet_manager.cc:1428] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:18.227942 12828 tablet_replica.cc:330] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: stopping tablet replica
I20260619 14:57:18.228044 12779 ts_tablet_manager.cc:1428] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:18.228175 12828 raft_consensus.cc:2229] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:18.228987 12780 tablet_bootstrap.cc:492] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: No bootstrap required, opened a new log
I20260619 14:57:18.229061 12780 ts_tablet_manager.cc:1397] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:57:18.229511 12780 raft_consensus.cc:348] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.229614 12780 raft_consensus.cc:374] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:18.229648 12780 raft_consensus.cc:729] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016935878ce14659a476d2e05f851a91, State: Initialized, Role: FOLLOWER
I20260619 14:57:18.229768 12780 consensus_queue.cc:260] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.229872 12828 raft_consensus.cc:2258] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:18.230000 12780 ts_tablet_manager.cc:1428] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:18.230198 12829 tablet_replica.cc:330] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: stopping tablet replica
I20260619 14:57:18.230469 12829 raft_consensus.cc:2229] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:18.230609 12829 raft_consensus.cc:2258] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:18.231278 12828 ts_tablet_manager.cc:1905] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:18.231571 12829 ts_tablet_manager.cc:1905] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:18.234150 12828 ts_tablet_manager.cc:1918] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:18.234165 12829 ts_tablet_manager.cc:1918] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:18.234237 12829 log.cc:1196] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/767c6625c3934aa5b00384934bd848d7
I20260619 14:57:18.234237 12828 log.cc:1196] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/767c6625c3934aa5b00384934bd848d7
I20260619 14:57:18.234498 12828 ts_tablet_manager.cc:1939] T 767c6625c3934aa5b00384934bd848d7 P 016935878ce14659a476d2e05f851a91: Deleting consensus metadata
I20260619 14:57:18.234498 12829 ts_tablet_manager.cc:1939] T 767c6625c3934aa5b00384934bd848d7 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting consensus metadata
I20260619 14:57:18.235334 12257 catalog_manager.cc:4841] TS 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397): tablet 767c6625c3934aa5b00384934bd848d7 (table test-table [id=e5fd98aef84b4433bd63d3cf5fd403c5]) successfully deleted
I20260619 14:57:18.235373 12259 catalog_manager.cc:4841] TS 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903): tablet 767c6625c3934aa5b00384934bd848d7 (table test-table [id=e5fd98aef84b4433bd63d3cf5fd403c5]) successfully deleted
I20260619 14:57:18.361733 12786 raft_consensus.cc:482] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:18.361920 12786 raft_consensus.cc:504] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.362417 12786 leader_election.cc:288] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903), 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:18.362762 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e17d3ea29843cc8a4371f3a0112211" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016935878ce14659a476d2e05f851a91" is_pre_election: true
I20260619 14:57:18.362946 12430 raft_consensus.cc:2454] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 0.
I20260619 14:57:18.362855 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e17d3ea29843cc8a4371f3a0112211" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659" is_pre_election: true
I20260619 14:57:18.363071 12710 raft_consensus.cc:2454] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 0.
I20260619 14:57:18.363286 12503 leader_election.cc:302] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:18.363488 12786 raft_consensus.cc:2781] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:18.363557 12786 raft_consensus.cc:482] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:18.363618 12786 raft_consensus.cc:3037] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:18.364602 12786 raft_consensus.cc:504] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.364915 12786 leader_election.cc:288] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 election: Requested vote from peers 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903), 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:18.365191 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e17d3ea29843cc8a4371f3a0112211" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659"
I20260619 14:57:18.365243 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e17d3ea29843cc8a4371f3a0112211" candidate_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016935878ce14659a476d2e05f851a91"
I20260619 14:57:18.365304 12710 raft_consensus.cc:3037] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:18.365379 12430 raft_consensus.cc:3037] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:18.366123 12710 raft_consensus.cc:2454] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 1.
I20260619 14:57:18.366313 12430 raft_consensus.cc:2454] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63207c115b1c45b1a6e8c0f3c6f249f1 in term 1.
I20260619 14:57:18.366365 12501 leader_election.cc:302] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cafa50e9f2f4202867968dc0954b659, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:18.366510 12786 raft_consensus.cc:2781] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:18.366664 12786 raft_consensus.cc:686] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 LEADER]: Becoming Leader. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Running, Role: LEADER
I20260619 14:57:18.366801 12786 consensus_queue.cc:237] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:18.368690 12272 catalog_manager.cc:5495] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130). New cstate: current_term: 1 leader_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:18.385887 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.383327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.10.71.130:43397)
0619 14:57:18.383428 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.383434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.383483 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:57:18.383584 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.383586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.383620 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.383694 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:18.383871 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.384415 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.385156 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.385343 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.385412 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.385599 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.385654 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.385702 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.385768 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.385805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:57:18.393321 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.390641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:51097 (local address 127.10.71.130:43397)
0619 14:57:18.390744 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:18.390754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:18.390765 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:18.390798 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:18.390799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:18.390833 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:18.390913 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:18.391100 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.391502 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.392289 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.392635 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.392727 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.392954 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.393014 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.393098 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:18.393175 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:57:18.393206 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:57:18.393321 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:18.390517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.10.71.130:51097)
0619 14:57:18.390662 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:18.390673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:18.390730 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:18.390931 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:18.390932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:18.390937 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:18.391011 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.391015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.391594 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.391597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.392185 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:18.392196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:18.392737 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:18.392740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:18.392815 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:18.392833 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:18.393110 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:18.393116 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:57:18.393169 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 14:57:18.393229 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:57:18.398010 12710 raft_consensus.cc:1261] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Refusing update from remote peer 63207c115b1c45b1a6e8c0f3c6f249f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:18.398010 12430 raft_consensus.cc:1261] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Refusing update from remote peer 63207c115b1c45b1a6e8c0f3c6f249f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:18.398381 12812 consensus_queue.cc:1035] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:18.398500 12786 consensus_queue.cc:1035] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:57:18.410758 12807 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:57:19.958889 12271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:19.959035 12271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:19.961153 12271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Sending DeleteTablet for 3 replicas of tablet f5e17d3ea29843cc8a4371f3a0112211
I20260619 14:57:19.961663 12410 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e17d3ea29843cc8a4371f3a0112211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:19 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:43554
I20260619 14:57:19.961892 12690 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e17d3ea29843cc8a4371f3a0112211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:19 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260619 14:57:19.962248 12550 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e17d3ea29843cc8a4371f3a0112211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:19 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260619 14:57:19.962572 12860 tablet_replica.cc:330] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: stopping tablet replica
I20260619 14:57:19.962731 12860 raft_consensus.cc:2229] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:19.963018 12860 raft_consensus.cc:2258] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:19.963724 12860 ts_tablet_manager.cc:1905] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:19.964236 12859 tablet_replica.cc:330] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: stopping tablet replica
I20260619 14:57:19.964375 12859 raft_consensus.cc:2229] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:19.964493 12859 raft_consensus.cc:2258] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:19.964985 12858 tablet_replica.cc:330] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: stopping tablet replica
I20260619 14:57:19.965173 12858 raft_consensus.cc:2229] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:19.965298 12858 raft_consensus.cc:2258] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:19.965363 12859 ts_tablet_manager.cc:1905] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:19.965803 12858 ts_tablet_manager.cc:1905] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:19.966635 12860 ts_tablet_manager.cc:1918] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:57:19.966722 12860 log.cc:1196] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/f5e17d3ea29843cc8a4371f3a0112211
I20260619 14:57:19.967082 12860 ts_tablet_manager.cc:1939] T f5e17d3ea29843cc8a4371f3a0112211 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting consensus metadata
I20260619 14:57:19.967736 12257 catalog_manager.cc:4841] TS 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397): tablet f5e17d3ea29843cc8a4371f3a0112211 (table test-table [id=2939844a09064918907e54d61d91c39c]) successfully deleted
I20260619 14:57:19.968025 12859 ts_tablet_manager.cc:1918] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:57:19.968050 12858 ts_tablet_manager.cc:1918] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:57:19.968111 12859 log.cc:1196] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/f5e17d3ea29843cc8a4371f3a0112211
I20260619 14:57:19.968124 12858 log.cc:1196] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/f5e17d3ea29843cc8a4371f3a0112211
I20260619 14:57:19.968441 12858 ts_tablet_manager.cc:1939] T f5e17d3ea29843cc8a4371f3a0112211 P 016935878ce14659a476d2e05f851a91: Deleting consensus metadata
I20260619 14:57:19.968649 12859 ts_tablet_manager.cc:1939] T f5e17d3ea29843cc8a4371f3a0112211 P 5cafa50e9f2f4202867968dc0954b659: Deleting consensus metadata
I20260619 14:57:19.969045 12259 catalog_manager.cc:4841] TS 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903): tablet f5e17d3ea29843cc8a4371f3a0112211 (table test-table [id=2939844a09064918907e54d61d91c39c]) successfully deleted
I20260619 14:57:19.969355 12257 catalog_manager.cc:4841] TS 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153): tablet f5e17d3ea29843cc8a4371f3a0112211 (table test-table [id=2939844a09064918907e54d61d91c39c]) successfully deleted
Jun 19 14:57:19 dist-test-slave-3qdh krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881039, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:57:19 dist-test-slave-3qdh krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881039, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:19.995044 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:19.982622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.10.71.190:45057)
0619 14:57:19.982898 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:19.982909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:19.982925 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:57:19.983103 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:19.983107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:19.983165 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:19.983258 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:19.983979 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:19.984589 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:19.985400 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:19.985627 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:19.985736 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:19.991728 (+ 5992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:19.991732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:19.991734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:19.991757 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:19.993340 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:19.993759 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:19.993764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:19.993766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:19.993853 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:19.994127 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:19.994131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:19.994133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:19.994326 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:19.994467 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:19.994727 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 14:57:19.994874 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260619 14:57:19.997617 12271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:19.998060 12271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:20.002954 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 32d1cb09c05a4de9b6754fc9371f1c0d (DEFAULT_TABLE table=test-table [id=30ea8ec260f5427b83639c3065012e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:20.003000 12550 tablet_service.cc:1467] Processing CreateTablet for tablet 32d1cb09c05a4de9b6754fc9371f1c0d (DEFAULT_TABLE table=test-table [id=30ea8ec260f5427b83639c3065012e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:20.003077 12690 tablet_service.cc:1467] Processing CreateTablet for tablet 32d1cb09c05a4de9b6754fc9371f1c0d (DEFAULT_TABLE table=test-table [id=30ea8ec260f5427b83639c3065012e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:20.003257 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d1cb09c05a4de9b6754fc9371f1c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:20.003312 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d1cb09c05a4de9b6754fc9371f1c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:20.003310 12550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d1cb09c05a4de9b6754fc9371f1c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:20.006215 12869 tablet_bootstrap.cc:492] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Bootstrap starting.
I20260619 14:57:20.006249 12871 tablet_bootstrap.cc:492] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Bootstrap starting.
I20260619 14:57:20.007097 12871 tablet_bootstrap.cc:654] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:20.007098 12869 tablet_bootstrap.cc:654] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:20.007790 12870 tablet_bootstrap.cc:492] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Bootstrap starting.
I20260619 14:57:20.008252 12869 tablet_bootstrap.cc:492] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: No bootstrap required, opened a new log
I20260619 14:57:20.008365 12869 ts_tablet_manager.cc:1397] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:20.008690 12870 tablet_bootstrap.cc:654] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:20.008816 12869 raft_consensus.cc:348] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.008916 12869 raft_consensus.cc:374] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:20.008956 12869 raft_consensus.cc:729] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cafa50e9f2f4202867968dc0954b659, State: Initialized, Role: FOLLOWER
I20260619 14:57:20.009085 12869 consensus_queue.cc:260] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.009497 12869 ts_tablet_manager.cc:1428] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:20.009864 12870 tablet_bootstrap.cc:492] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: No bootstrap required, opened a new log
I20260619 14:57:20.010000 12870 ts_tablet_manager.cc:1397] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:20.010468 12870 raft_consensus.cc:348] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.010614 12870 raft_consensus.cc:374] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:20.010702 12870 raft_consensus.cc:729] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Initialized, Role: FOLLOWER
I20260619 14:57:20.010876 12870 consensus_queue.cc:260] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.011262 12870 ts_tablet_manager.cc:1428] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:57:20.011765 12871 tablet_bootstrap.cc:492] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: No bootstrap required, opened a new log
I20260619 14:57:20.011871 12871 ts_tablet_manager.cc:1397] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 14:57:20.012254 12871 raft_consensus.cc:348] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.012426 12871 raft_consensus.cc:374] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:20.012497 12871 raft_consensus.cc:729] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016935878ce14659a476d2e05f851a91, State: Initialized, Role: FOLLOWER
I20260619 14:57:20.012653 12871 consensus_queue.cc:260] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.013031 12871 ts_tablet_manager.cc:1428] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:20.033885 12876 raft_consensus.cc:482] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:20.033977 12876 raft_consensus.cc:504] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.034297 12876 leader_election.cc:288] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397), 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:20.034653 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d1cb09c05a4de9b6754fc9371f1c0d" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" is_pre_election: true
I20260619 14:57:20.034759 12570 raft_consensus.cc:2454] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 0.
I20260619 14:57:20.034754 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d1cb09c05a4de9b6754fc9371f1c0d" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659" is_pre_election: true
I20260619 14:57:20.034898 12709 raft_consensus.cc:2454] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 0.
I20260619 14:57:20.035116 12361 leader_election.cc:302] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:20.035266 12876 raft_consensus.cc:2781] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:20.035319 12876 raft_consensus.cc:482] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:20.035344 12876 raft_consensus.cc:3037] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:20.036358 12876 raft_consensus.cc:504] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.036769 12876 leader_election.cc:288] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 election: Requested vote from peers 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397), 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153)
I20260619 14:57:20.037009 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d1cb09c05a4de9b6754fc9371f1c0d" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1"
I20260619 14:57:20.037127 12570 raft_consensus.cc:3037] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:20.037115 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d1cb09c05a4de9b6754fc9371f1c0d" candidate_uuid: "016935878ce14659a476d2e05f851a91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cafa50e9f2f4202867968dc0954b659"
I20260619 14:57:20.037221 12709 raft_consensus.cc:3037] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:20.038020 12570 raft_consensus.cc:2454] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 1.
I20260619 14:57:20.038218 12709 raft_consensus.cc:2454] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016935878ce14659a476d2e05f851a91 in term 1.
I20260619 14:57:20.038323 12361 leader_election.cc:302] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:20.038527 12876 raft_consensus.cc:2781] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:20.038753 12876 raft_consensus.cc:686] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 1 LEADER]: Becoming Leader. State: Replica: 016935878ce14659a476d2e05f851a91, State: Running, Role: LEADER
I20260619 14:57:20.038931 12876 consensus_queue.cc:237] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.040421 12272 catalog_manager.cc:5495] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 016935878ce14659a476d2e05f851a91 (127.10.71.129). New cstate: current_term: 1 leader_uuid: "016935878ce14659a476d2e05f851a91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:20.054198 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.051361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.10.71.129:41903)
0619 14:57:20.051611 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:20.051618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:20.051629 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:20.051666 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:20.051668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:20.051704 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:20.051775 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:20.051959 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.052537 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.053381 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.053593 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.053666 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.053874 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:20.053933 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:20.053983 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:20.054056 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:57:20.054095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":114,"threads_started":1}
I20260619 14:57:20.060092 12709 raft_consensus.cc:1261] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Refusing update from remote peer 016935878ce14659a476d2e05f851a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:20.060107 12570 raft_consensus.cc:1261] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Refusing update from remote peer 016935878ce14659a476d2e05f851a91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:20.060473 12877 consensus_queue.cc:1035] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:20.060606 12876 consensus_queue.cc:1035] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:20.086982 12273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:20.087139 12273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:20.088986 12273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Sending DeleteTablet for 3 replicas of tablet 32d1cb09c05a4de9b6754fc9371f1c0d
I20260619 14:57:20.089474 12550 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d1cb09c05a4de9b6754fc9371f1c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:20 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260619 14:57:20.089581 12690 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d1cb09c05a4de9b6754fc9371f1c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:20 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260619 14:57:20.089577 12410 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d1cb09c05a4de9b6754fc9371f1c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:20 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:43554
I20260619 14:57:20.089660 12860 tablet_replica.cc:330] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: stopping tablet replica
I20260619 14:57:20.089730 12858 tablet_replica.cc:330] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: stopping tablet replica
I20260619 14:57:20.089730 12859 tablet_replica.cc:330] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: stopping tablet replica
I20260619 14:57:20.089793 12860 raft_consensus.cc:2229] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:20.089831 12858 raft_consensus.cc:2229] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:20.089831 12859 raft_consensus.cc:2229] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:20.089915 12860 raft_consensus.cc:2258] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:20.089924 12859 raft_consensus.cc:2258] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:20.089985 12858 raft_consensus.cc:2258] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:20.090579 12859 ts_tablet_manager.cc:1905] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:20.090613 12858 ts_tablet_manager.cc:1905] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:20.090907 12273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:20.091238 12273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:20.092569 12858 ts_tablet_manager.cc:1918] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:20.092653 12858 log.cc:1196] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/32d1cb09c05a4de9b6754fc9371f1c0d
I20260619 14:57:20.092896 12858 ts_tablet_manager.cc:1939] T 32d1cb09c05a4de9b6754fc9371f1c0d P 016935878ce14659a476d2e05f851a91: Deleting consensus metadata
I20260619 14:57:20.093093 12859 ts_tablet_manager.cc:1918] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:20.093160 12859 log.cc:1196] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/32d1cb09c05a4de9b6754fc9371f1c0d
I20260619 14:57:20.093374 12259 catalog_manager.cc:4841] TS 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903): tablet 32d1cb09c05a4de9b6754fc9371f1c0d (table test-table [id=30ea8ec260f5427b83639c3065012e09]) successfully deleted
I20260619 14:57:20.093429 12859 ts_tablet_manager.cc:1939] T 32d1cb09c05a4de9b6754fc9371f1c0d P 5cafa50e9f2f4202867968dc0954b659: Deleting consensus metadata
I20260619 14:57:20.093994 12257 catalog_manager.cc:4841] TS 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153): tablet 32d1cb09c05a4de9b6754fc9371f1c0d (table test-table [id=30ea8ec260f5427b83639c3065012e09]) successfully deleted
I20260619 14:57:20.094771 12860 ts_tablet_manager.cc:1905] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:20.095742 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 3f79c9bb3bd64df1a21fda00791fd392 (DEFAULT_TABLE table=test-table [id=dba80ed8783c4adbbae4169f9405ca8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:20.095944 12550 tablet_service.cc:1467] Processing CreateTablet for tablet 3f79c9bb3bd64df1a21fda00791fd392 (DEFAULT_TABLE table=test-table [id=dba80ed8783c4adbbae4169f9405ca8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:20.095994 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f79c9bb3bd64df1a21fda00791fd392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:20.096040 12690 tablet_service.cc:1467] Processing CreateTablet for tablet 3f79c9bb3bd64df1a21fda00791fd392 (DEFAULT_TABLE table=test-table [id=dba80ed8783c4adbbae4169f9405ca8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:20.096213 12550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f79c9bb3bd64df1a21fda00791fd392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:20.096246 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f79c9bb3bd64df1a21fda00791fd392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:20.097024 12860 ts_tablet_manager.cc:1918] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:20.097083 12860 log.cc:1196] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/32d1cb09c05a4de9b6754fc9371f1c0d
I20260619 14:57:20.097335 12860 ts_tablet_manager.cc:1939] T 32d1cb09c05a4de9b6754fc9371f1c0d P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting consensus metadata
I20260619 14:57:20.098014 12871 tablet_bootstrap.cc:492] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91: Bootstrap starting.
I20260619 14:57:20.098104 12869 tablet_bootstrap.cc:492] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Bootstrap starting.
I20260619 14:57:20.098304 12257 catalog_manager.cc:4841] TS 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397): tablet 32d1cb09c05a4de9b6754fc9371f1c0d (table test-table [id=30ea8ec260f5427b83639c3065012e09]) successfully deleted
I20260619 14:57:20.098853 12871 tablet_bootstrap.cc:654] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:20.099051 12869 tablet_bootstrap.cc:654] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:20.099696 12870 tablet_bootstrap.cc:492] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Bootstrap starting.
I20260619 14:57:20.099962 12871 tablet_bootstrap.cc:492] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91: No bootstrap required, opened a new log
I20260619 14:57:20.100062 12871 ts_tablet_manager.cc:1397] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:20.100252 12869 tablet_bootstrap.cc:492] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: No bootstrap required, opened a new log
I20260619 14:57:20.100363 12869 ts_tablet_manager.cc:1397] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 14:57:20.100471 12871 raft_consensus.cc:348] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.100598 12871 raft_consensus.cc:374] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:20.100636 12871 raft_consensus.cc:729] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016935878ce14659a476d2e05f851a91, State: Initialized, Role: FOLLOWER
I20260619 14:57:20.100776 12870 tablet_bootstrap.cc:654] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:20.100767 12871 consensus_queue.cc:260] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.100812 12869 raft_consensus.cc:348] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.100898 12869 raft_consensus.cc:374] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:20.100927 12869 raft_consensus.cc:729] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cafa50e9f2f4202867968dc0954b659, State: Initialized, Role: FOLLOWER
I20260619 14:57:20.101024 12871 ts_tablet_manager.cc:1428] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:20.101035 12869 consensus_queue.cc:260] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.101351 12869 ts_tablet_manager.cc:1428] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:20.101718 12870 tablet_bootstrap.cc:492] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: No bootstrap required, opened a new log
I20260619 14:57:20.101789 12870 ts_tablet_manager.cc:1397] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:20.102205 12870 raft_consensus.cc:348] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.102316 12870 raft_consensus.cc:374] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:20.102345 12870 raft_consensus.cc:729] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63207c115b1c45b1a6e8c0f3c6f249f1, State: Initialized, Role: FOLLOWER
I20260619 14:57:20.102456 12870 consensus_queue.cc:260] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.102722 12870 ts_tablet_manager.cc:1428] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:20.237469 12875 raft_consensus.cc:482] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:20.237671 12875 raft_consensus.cc:504] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.238854 12875 leader_election.cc:288] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903), 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397)
I20260619 14:57:20.244302 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.239170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:49181 (local address 127.10.71.129:41903)
0619 14:57:20.239293 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:20.239298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:20.239460 (+ 162us) server_negotiation.cc:407] Connection header received
0619 14:57:20.239631 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:20.239635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:20.239684 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:20.239769 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:20.240218 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.241030 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.242947 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.243590 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.243709 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.243761 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:20.244097 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 14:57:20.244162 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:57:20.244305 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.239207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:56219 (local address 127.10.71.130:43397)
0619 14:57:20.239459 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:20.239474 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:20.239508 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:57:20.239632 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:20.239635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:20.239685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:20.239770 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:20.240218 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.241041 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.242947 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.243601 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.243709 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.243761 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:20.244081 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 14:57:20.244155 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260619 14:57:20.244367 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.239125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.10.71.131:56219)
0619 14:57:20.239441 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:20.239453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:20.239531 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:20.239799 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:20.239801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:20.239841 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:20.240067 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.240079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.241183 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.241187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.242818 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.242830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.243713 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.243718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.243916 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.243977 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 14:57:20.244049 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:57:20.244141 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":361,"spinlock_wait_cycles":29312,"thread_start_us":124,"threads_started":1}
I20260619 14:57:20.244560 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.239031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:41903 (local address 127.10.71.131:49181)
0619 14:57:20.239349 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:20.239364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:20.239490 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:20.239799 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:20.239801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:20.239842 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:20.240077 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.240082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.241183 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.241187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.242818 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.242830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.243713 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.243718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.243911 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.243969 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 14:57:20.244050 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:57:20.244138 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":256,"spinlock_wait_cycles":6272,"thread_start_us":70,"threads_started":1}
I20260619 14:57:20.245039 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f79c9bb3bd64df1a21fda00791fd392" candidate_uuid: "5cafa50e9f2f4202867968dc0954b659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016935878ce14659a476d2e05f851a91" is_pre_election: true
I20260619 14:57:20.245097 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f79c9bb3bd64df1a21fda00791fd392" candidate_uuid: "5cafa50e9f2f4202867968dc0954b659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" is_pre_election: true
I20260619 14:57:20.245195 12430 raft_consensus.cc:2454] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cafa50e9f2f4202867968dc0954b659 in term 0.
I20260619 14:57:20.245229 12570 raft_consensus.cc:2454] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cafa50e9f2f4202867968dc0954b659 in term 0.
I20260619 14:57:20.245558 12643 leader_election.cc:302] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016935878ce14659a476d2e05f851a91, 5cafa50e9f2f4202867968dc0954b659; no voters:
I20260619 14:57:20.245808 12875 raft_consensus.cc:2781] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:20.245868 12875 raft_consensus.cc:482] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:20.245906 12875 raft_consensus.cc:3037] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:20.246925 12875 raft_consensus.cc:504] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.247233 12875 leader_election.cc:288] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [CANDIDATE]: Term 1 election: Requested vote from peers 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903), 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397)
I20260619 14:57:20.247515 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f79c9bb3bd64df1a21fda00791fd392" candidate_uuid: "5cafa50e9f2f4202867968dc0954b659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016935878ce14659a476d2e05f851a91"
I20260619 14:57:20.247582 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f79c9bb3bd64df1a21fda00791fd392" candidate_uuid: "5cafa50e9f2f4202867968dc0954b659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1"
I20260619 14:57:20.247641 12430 raft_consensus.cc:3037] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:20.247705 12570 raft_consensus.cc:3037] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:20.248811 12570 raft_consensus.cc:2454] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cafa50e9f2f4202867968dc0954b659 in term 1.
I20260619 14:57:20.248809 12430 raft_consensus.cc:2454] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cafa50e9f2f4202867968dc0954b659 in term 1.
I20260619 14:57:20.249069 12641 leader_election.cc:302] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cafa50e9f2f4202867968dc0954b659, 63207c115b1c45b1a6e8c0f3c6f249f1; no voters:
I20260619 14:57:20.249320 12875 raft_consensus.cc:2781] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:20.249493 12875 raft_consensus.cc:686] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 1 LEADER]: Becoming Leader. State: Replica: 5cafa50e9f2f4202867968dc0954b659, State: Running, Role: LEADER
I20260619 14:57:20.249716 12875 consensus_queue.cc:237] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } }
I20260619 14:57:20.251526 12273 catalog_manager.cc:5495] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "5cafa50e9f2f4202867968dc0954b659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cafa50e9f2f4202867968dc0954b659" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36153 } health_report { overall_health: HEALTHY } } }
I20260619 14:57:20.296672 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.293491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.10.71.131:36153)
0619 14:57:20.293721 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:20.293728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:20.293739 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:20.293793 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:20.293795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:20.293838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:20.293930 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:20.294103 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.294618 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.295310 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.295624 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.295694 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.295865 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:20.296254 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:20.296413 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:20.296441 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 14:57:20.296479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
I20260619 14:57:20.305330 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.302429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:45675 (local address 127.10.71.130:43397)
0619 14:57:20.302538 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:20.302542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:20.302656 (+ 114us) server_negotiation.cc:407] Connection header received
0619 14:57:20.302799 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:20.302802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:20.302843 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:20.302929 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:20.303175 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.303607 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.304392 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.304601 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.304705 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.304964 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:20.305028 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:20.305106 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:20.305181 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 14:57:20.305222 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:57:20.305371 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.302248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43397 (local address 127.10.71.131:45675)
0619 14:57:20.302556 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:20.302584 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:20.302700 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:20.302937 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:20.302939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:20.302947 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:20.303054 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.303065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.303702 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.303707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.304244 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.304252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.304718 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.304721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.304816 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.304847 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:20.305108 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:20.305116 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:57:20.305173 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:57:20.305234 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260619 14:57:20.310067 12570 raft_consensus.cc:1261] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Refusing update from remote peer 5cafa50e9f2f4202867968dc0954b659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:20.310067 12430 raft_consensus.cc:1261] T 3f79c9bb3bd64df1a21fda00791fd392 P 016935878ce14659a476d2e05f851a91 [term 1 FOLLOWER]: Refusing update from remote peer 5cafa50e9f2f4202867968dc0954b659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:20.310509 12891 consensus_queue.cc:1035] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016935878ce14659a476d2e05f851a91" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:20.310693 12875 consensus_queue.cc:1035] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63207c115b1c45b1a6e8c0f3c6f249f1" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:20.326354 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.323283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:38839 (local address 127.10.71.131:36153)
0619 14:57:20.323391 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:20.323395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:20.323592 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:57:20.323663 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:20.323668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:20.323713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:20.323797 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:20.324000 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.324595 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.325406 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.325611 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.325708 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.325941 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:20.326003 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:20.326120 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:20.326204 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:57:20.326248 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:20.326390 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:20.323233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36153 (local address 127.10.71.130:38839)
0619 14:57:20.323505 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:20.323516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:20.323603 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:20.323806 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:20.323808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:20.323815 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:20.323903 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.323908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.324711 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.324716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.325303 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:20.325311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:20.325724 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:20.325727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:20.325811 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:20.325838 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:20.326122 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:20.326126 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:57:20.326183 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:57:20.326250 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
W20260619 14:57:20.327163 12842 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:57:21.882433 12271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:21.882587 12271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:21.884783 12271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fe1e6161cb44c6bc991e60bedb8c6d: Sending DeleteTablet for 3 replicas of tablet 3f79c9bb3bd64df1a21fda00791fd392
I20260619 14:57:21.885296 12690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f79c9bb3bd64df1a21fda00791fd392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:21 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260619 14:57:21.885300 12410 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f79c9bb3bd64df1a21fda00791fd392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:21 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:43554
I20260619 14:57:21.885623 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 12345
I20260619 14:57:21.885627 12925 tablet_replica.cc:330] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: stopping tablet replica
I20260619 14:57:21.886111 12550 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f79c9bb3bd64df1a21fda00791fd392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:21 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260619 14:57:21.886313 12925 raft_consensus.cc:2229] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:21.886417 12926 tablet_replica.cc:330] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: stopping tablet replica
I20260619 14:57:21.886525 12926 raft_consensus.cc:2229] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:21.886608 12926 raft_consensus.cc:2258] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:21.886616 12925 raft_consensus.cc:2258] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:21.887228 12925 ts_tablet_manager.cc:1905] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:21.887225 12926 ts_tablet_manager.cc:1905] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:21.889791 12925 ts_tablet_manager.cc:1918] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:57:21.889791 12926 ts_tablet_manager.cc:1918] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:57:21.889870 12926 log.cc:1196] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/3f79c9bb3bd64df1a21fda00791fd392
I20260619 14:57:21.889870 12925 log.cc:1196] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/3f79c9bb3bd64df1a21fda00791fd392
I20260619 14:57:21.890152 12926 ts_tablet_manager.cc:1939] T 3f79c9bb3bd64df1a21fda00791fd392 P 63207c115b1c45b1a6e8c0f3c6f249f1: Deleting consensus metadata
I20260619 14:57:21.890158 12925 ts_tablet_manager.cc:1939] T 3f79c9bb3bd64df1a21fda00791fd392 P 5cafa50e9f2f4202867968dc0954b659: Deleting consensus metadata
I20260619 14:57:21.890717 12257 catalog_manager.cc:4841] TS 63207c115b1c45b1a6e8c0f3c6f249f1 (127.10.71.130:43397): tablet 3f79c9bb3bd64df1a21fda00791fd392 (table test-table [id=dba80ed8783c4adbbae4169f9405ca8c]) successfully deleted
I20260619 14:57:21.890884 12257 catalog_manager.cc:4841] TS 5cafa50e9f2f4202867968dc0954b659 (127.10.71.131:36153): tablet 3f79c9bb3bd64df1a21fda00791fd392 (table test-table [id=dba80ed8783c4adbbae4169f9405ca8c]) successfully deleted
W20260619 14:57:21.893730 12259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:41903 (error 108)
I20260619 14:57:21.893952 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 12485
I20260619 14:57:21.895540 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:21.895185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:41903 (local address 127.0.0.1:43578)
0619 14:57:21.895402 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:21.895466 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
W20260619 14:57:21.895769 12259 catalog_manager.cc:4584] TS 016935878ce14659a476d2e05f851a91 (127.10.71.129:41903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f79c9bb3bd64df1a21fda00791fd392: Network error: Client connection negotiation failed: client connection to 127.10.71.129:41903: connect: Connection refused (error 111)
I20260619 14:57:21.901083 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 12625
I20260619 14:57:21.907315 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 12241
2026-06-19T14:57:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6132 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:21 dist-test-slave-3qdh krb5kdc[12931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:21 dist-test-slave-3qdh krb5kdc[12931](info): set up 2 sockets
Jun 19 14:57:21 dist-test-slave-3qdh krb5kdc[12931](info): commencing operation
krb5kdc: starting...
W20260619 14:57:22.887827 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:22 dist-test-slave-3qdh krb5kdc[12931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:22.995010 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:35881
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:35881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:23.095177 12944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:23.095396 12944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:23.095454 12944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:23.098841 12944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:23.098909 12944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:23.098943 12944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:23.098994 12944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:23.099025 12944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:23.103358 12944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:35881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:23.104207 12944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:23.105098 12944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:23.112542 12950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.112648 12944 server_base.cc:1034] running on GCE node
W20260619 14:57:23.112525 12949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:23.112798 12952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.113327 12944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:23.114189 12944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:23.115340 12944 hybrid_clock.cc:648] HybridClock initialized: now 1781881043115317 us; error 34 us; skew 500 ppm
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:23.117805 12944 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:23.118492 12944 webserver.cc:460] Webserver started at http://127.10.71.190:40131/ using document root <none> and password file <none>
I20260619 14:57:23.118934 12944 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:23.118985 12944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:23.119158 12944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:23.120790 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "84050f0a50d74f85baabdf5a644c0c20"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "4177426dc9e92fddd5b8c8a2addd8662"
server_key_iv: "5a245211c2a1183ce417b59358a56015"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.121289 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "84050f0a50d74f85baabdf5a644c0c20"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "4177426dc9e92fddd5b8c8a2addd8662"
server_key_iv: "5a245211c2a1183ce417b59358a56015"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.124478 12944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:23.126292 12959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.127290 12944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:23.127409 12944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "84050f0a50d74f85baabdf5a644c0c20"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "4177426dc9e92fddd5b8c8a2addd8662"
server_key_iv: "5a245211c2a1183ce417b59358a56015"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.127503 12944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:23.159580 12944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:23.161932 12944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:23.162124 12944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:23.168609 12944 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:35881
I20260619 14:57:23.168617 13011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:35881 every 8 connection(s)
I20260619 14:57:23.169451 12944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:23.170081 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 12944
I20260619 14:57:23.170174 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:23.170396 10526 external_mini_cluster.cc:1428] Setting key 6b5d6847e3c305f7ff92e28887f7ac48
I20260619 14:57:23.172204 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:23.177841 13012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20: Bootstrap starting.
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:23.180006 13012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:23.180684 13012 log.cc:824] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:23.182152 13012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20: No bootstrap required, opened a new log
I20260619 14:57:23.183110 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.171609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.10.71.190:35881)
0619 14:57:23.171996 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:23.172010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:23.172039 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:57:23.172717 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:23.172733 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:23.173014 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:23.173271 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:23.174122 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.174978 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.175950 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.176428 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.176536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.179028 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:23.179033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:23.179043 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:23.179068 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:23.180699 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.181100 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.181104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.181107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.181198 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.181455 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.181459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.181461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.181765 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:23.181930 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:23.182222 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 14:57:23.182516 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260619 14:57:23.185689 13012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050f0a50d74f85baabdf5a644c0c20" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35881 } }
I20260619 14:57:23.185899 13012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:23.185981 13012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84050f0a50d74f85baabdf5a644c0c20, State: Initialized, Role: FOLLOWER
I20260619 14:57:23.186344 13012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050f0a50d74f85baabdf5a644c0c20" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35881 } }
I20260619 14:57:23.186483 13012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:23.186563 13012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:23.186676 13012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:23.187438 13012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050f0a50d74f85baabdf5a644c0c20" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35881 } }
I20260619 14:57:23.187747 13012 leader_election.cc:302] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84050f0a50d74f85baabdf5a644c0c20; no voters:
I20260619 14:57:23.188037 13012 leader_election.cc:288] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:23.188194 13017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:23.188488 13017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [term 1 LEADER]: Becoming Leader. State: Replica: 84050f0a50d74f85baabdf5a644c0c20, State: Running, Role: LEADER
I20260619 14:57:23.188760 13017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050f0a50d74f85baabdf5a644c0c20" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35881 } }
I20260619 14:57:23.189074 13012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:23.190311 13018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84050f0a50d74f85baabdf5a644c0c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050f0a50d74f85baabdf5a644c0c20" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35881 } } }
I20260619 14:57:23.190361 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84050f0a50d74f85baabdf5a644c0c20. Latest consensus state: current_term: 1 leader_uuid: "84050f0a50d74f85baabdf5a644c0c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84050f0a50d74f85baabdf5a644c0c20" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35881 } } }
I20260619 14:57:23.190471 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:23.190421 13018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:23.190959 13029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:23.193749 13029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:23.198289 13029 catalog_manager.cc:1347] Generated new cluster ID: 5247e3918a4448d3b0aae1bee256b928
I20260619 14:57:23.198377 13029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:23.211501 13029 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:23.212234 13029 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:23.219185 13029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84050f0a50d74f85baabdf5a644c0c20: Generated new TSK 0
I20260619 14:57:23.219715 13029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:23.243395 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35881
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:23.344520 13038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:23.344830 13038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:23.344897 13038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:23.351107 13038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:23.351152 13038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:23.351203 13038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:23.355923 13038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:23.356791 13038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:23.357551 13038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:23.365933 13046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:23.365907 13044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.365949 13038 server_base.cc:1034] running on GCE node
W20260619 14:57:23.365900 13043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.366432 13038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:23.366950 13038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:23.368126 13038 hybrid_clock.cc:648] HybridClock initialized: now 1781881043368119 us; error 25 us; skew 500 ppm
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:23.370726 13038 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:23.371425 13038 webserver.cc:460] Webserver started at http://127.10.71.129:38019/ using document root <none> and password file <none>
I20260619 14:57:23.371897 13038 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:23.371975 13038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:23.372171 13038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:23.373984 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "4e1653d8df6b40538b056fe341a1c40c"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "f3da366cb3cd72dd54daacb975e8e721"
server_key_iv: "9763188c43a201e9f8409ded5a36b6cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.374477 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "4e1653d8df6b40538b056fe341a1c40c"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "f3da366cb3cd72dd54daacb975e8e721"
server_key_iv: "9763188c43a201e9f8409ded5a36b6cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.377636 13038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:23.379621 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.380645 13038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:23.380777 13038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "4e1653d8df6b40538b056fe341a1c40c"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "f3da366cb3cd72dd54daacb975e8e721"
server_key_iv: "9763188c43a201e9f8409ded5a36b6cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.380897 13038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:23.398351 13038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:23.400928 13038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:23.401094 13038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:23.401537 13038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:23.402271 13038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:23.402338 13038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.402406 13038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:23.402449 13038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.410465 13038 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:33217
I20260619 14:57:23.410501 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:33217 every 8 connection(s)
I20260619 14:57:23.411123 13038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:23.418708 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13038
I20260619 14:57:23.418826 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:23.419047 10526 external_mini_cluster.cc:1428] Setting key d9f01c4699e758f77ef086935fc2cd0b
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:23.422892 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.412606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:39613 (local address 127.10.71.190:35881)
0619 14:57:23.412730 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:23.412734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:23.413475 (+ 741us) server_negotiation.cc:407] Connection header received
0619 14:57:23.414732 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:23.414735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:23.414790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:23.414879 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:23.415812 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.416373 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.417358 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.417585 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.417681 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.419741 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:23.419745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:23.419747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:23.419771 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:23.421088 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.421520 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.421525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.421527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.421611 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.421912 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.421915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.421917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.422074 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:23.422167 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:23.422616 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 14:57:23.422736 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:23.423533 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.412873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35881 (local address 127.10.71.129:39613)
0619 14:57:23.413317 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:23.413366 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:23.414540 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:23.415016 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:23.415022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:23.415305 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:23.415668 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:23.415679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.416481 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.416484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:23.417237 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:23.417245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.417719 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.417722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:23.417865 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.418305 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:23.418317 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:23.419590 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:23.421193 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:23.421198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:23.421209 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:23.421426 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:23.421698 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:23.421701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:23.421703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:23.421820 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:23.422166 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:23.422172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:23.422396 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 14:57:23.422567 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 14:57:23.422839 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260619 14:57:23.424551 13167 heartbeater.cc:347] Connected to a master server at 127.10.71.190:35881
I20260619 14:57:23.424774 13167 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:23.425268 13167 heartbeater.cc:511] Master 127.10.71.190:35881 requested a full tablet report, sending...
I20260619 14:57:23.426720 12976 ts_manager.cc:194] Registered new tserver with Master: 4e1653d8df6b40538b056fe341a1c40c (127.10.71.129:33217)
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:23.428143 12976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:39613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:23.442842 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35881
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:23.536639 13172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:23.536818 13172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:23.536854 13172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:23.540555 13172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:23.540601 13172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:23.540689 13172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:23.545439 13172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:23.546202 13172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:23.547055 13172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:23.555380 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:23.555433 13180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.555526 13172 server_base.cc:1034] running on GCE node
W20260619 14:57:23.555466 13177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.555917 13172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:23.556563 13172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:23.557730 13172 hybrid_clock.cc:648] HybridClock initialized: now 1781881043557711 us; error 40 us; skew 500 ppm
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:23.560098 13172 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:23.560796 13172 webserver.cc:460] Webserver started at http://127.10.71.130:46185/ using document root <none> and password file <none>
I20260619 14:57:23.561195 13172 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:23.561266 13172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:23.561399 13172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:23.563097 13172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "04996210f6e245cc839ba9f7cb392f33"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "11441682a8446274f8170f9977904994"
server_key_iv: "de6179d4202491e9a1eb0c6ca9b29250"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.563633 13172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "04996210f6e245cc839ba9f7cb392f33"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "11441682a8446274f8170f9977904994"
server_key_iv: "de6179d4202491e9a1eb0c6ca9b29250"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.566831 13172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:57:23.569096 13187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.570137 13172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:23.570246 13172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "04996210f6e245cc839ba9f7cb392f33"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "11441682a8446274f8170f9977904994"
server_key_iv: "de6179d4202491e9a1eb0c6ca9b29250"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.570335 13172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:23.580675 13172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:23.583096 13172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:23.583313 13172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:23.583841 13172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:23.584638 13172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:23.584688 13172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.584749 13172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:23.584791 13172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.593492 13172 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:37701
I20260619 14:57:23.593571 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:37701 every 8 connection(s)
I20260619 14:57:23.594287 13172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:23.597474 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13172
I20260619 14:57:23.597587 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:23.597807 10526 external_mini_cluster.cc:1428] Setting key 3b6e3ca8826e485ed23d25b35dba63be
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:57:23.607239 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.595857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:37009 (local address 127.10.71.190:35881)
0619 14:57:23.595984 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:23.595987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:23.596683 (+ 696us) server_negotiation.cc:407] Connection header received
0619 14:57:23.598023 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:23.598026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:23.598089 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:23.598181 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:23.599384 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.600105 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.601167 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.601355 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.601452 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.603665 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:23.603668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:23.603670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:23.603689 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:23.605348 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.605821 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.605826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.605827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.605905 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.606206 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.606209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.606210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.606416 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:23.606504 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:23.606945 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 14:57:23.607045 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:57:23.607873 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.596144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35881 (local address 127.10.71.130:37009)
0619 14:57:23.596561 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:23.596595 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:23.597833 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:23.598338 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:23.598345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:23.598715 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:23.599251 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:23.599263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.600240 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.600242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:23.601053 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:23.601061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.601477 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.601480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:23.601621 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.602105 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:23.602119 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:23.603465 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:23.605449 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:23.605454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:23.605465 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:23.605696 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:23.605990 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:23.605993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:23.605995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:23.606115 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:23.606514 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:23.606521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:23.606747 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 14:57:23.606905 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:57:23.607153 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 14:57:23.608948 13301 heartbeater.cc:347] Connected to a master server at 127.10.71.190:35881
I20260619 14:57:23.609172 13301 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:23.609627 13301 heartbeater.cc:511] Master 127.10.71.190:35881 requested a full tablet report, sending...
I20260619 14:57:23.610600 12976 ts_manager.cc:194] Registered new tserver with Master: 04996210f6e245cc839ba9f7cb392f33 (127.10.71.130:37701)
I20260619 14:57:23.611164 12976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:37009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:23.622200 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35881
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:23.717090 13306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:23.717295 13306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:23.717358 13306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:23.720968 13306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:23.721028 13306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:23.721104 13306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:23.725605 13306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:23.726466 13306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:23.727263 13306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:23.735543 13314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:23.735571 13311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:23.735540 13312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:23.735836 13306 server_base.cc:1034] running on GCE node
I20260619 14:57:23.736168 13306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:23.736696 13306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:23.737864 13306 hybrid_clock.cc:648] HybridClock initialized: now 1781881043737853 us; error 30 us; skew 500 ppm
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:23.740140 13306 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:23.740918 13306 webserver.cc:460] Webserver started at http://127.10.71.131:43319/ using document root <none> and password file <none>
I20260619 14:57:23.741389 13306 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:23.741470 13306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:23.741680 13306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:23.743409 13306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "419d0cbfa70249d7827ad04630323e88"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "2b4501100fe52c2635dc53c5c402da33"
server_key_iv: "adbbf616349c2ab5a57c23d4d823d21f"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.743920 13306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "419d0cbfa70249d7827ad04630323e88"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "2b4501100fe52c2635dc53c5c402da33"
server_key_iv: "adbbf616349c2ab5a57c23d4d823d21f"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.747064 13306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:23.749219 13321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.750192 13306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:23.750308 13306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "419d0cbfa70249d7827ad04630323e88"
format_stamp: "Formatted at 2026-06-19 14:57:23 on dist-test-slave-3qdh"
server_key: "2b4501100fe52c2635dc53c5c402da33"
server_key_iv: "adbbf616349c2ab5a57c23d4d823d21f"
server_key_version: "encryptionkey@0"
I20260619 14:57:23.750401 13306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:23.762837 13306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:23.765127 13306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:23.765296 13306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:23.765784 13306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:23.766444 13306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:23.766510 13306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.766570 13306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:23.766628 13306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:23.774709 13306 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:35763
I20260619 14:57:23.774757 13434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:35763 every 8 connection(s)
I20260619 14:57:23.775399 13306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:23.776672 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13306
I20260619 14:57:23.776782 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:23.777115 10526 external_mini_cluster.cc:1428] Setting key 016f2b3a25cf060c1ff679efee28f019
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[12931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:23.786939 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.776898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:51379 (local address 127.10.71.190:35881)
0619 14:57:23.777108 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:23.777113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:23.777844 (+ 731us) server_negotiation.cc:407] Connection header received
0619 14:57:23.778999 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:23.779004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:23.779053 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:23.779124 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:23.780047 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.780655 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.781793 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.782000 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.782081 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.784183 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:23.784186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:23.784188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:23.784209 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:23.785501 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.785893 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.785895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.785896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.785958 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:23.786210 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:23.786211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:23.786212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:23.786351 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:23.786416 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:23.786763 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:57:23.786839 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:57:23.787560 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:23.777316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35881 (local address 127.10.71.131:51379)
0619 14:57:23.777717 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:23.777762 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:23.778833 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:23.779263 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:23.779269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:23.779551 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:23.779943 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:23.779953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.780855 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.780858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:23.781690 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:23.781699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:23.782114 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:23.782116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:23.782248 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:23.782704 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:23.782718 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:23.784052 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:23.785593 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:23.785597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:23.785606 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:23.785816 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:23.786025 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:23.786027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:23.786029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:23.786135 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:23.786427 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:23.786430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:23.786597 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 14:57:23.786723 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:57:23.786909 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":110,"threads_started":1}
I20260619 14:57:23.788595 13435 heartbeater.cc:347] Connected to a master server at 127.10.71.190:35881
I20260619 14:57:23.788801 13435 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:23.789338 13435 heartbeater.cc:511] Master 127.10.71.190:35881 requested a full tablet report, sending...
I20260619 14:57:23.790334 12976 ts_manager.cc:194] Registered new tserver with Master: 419d0cbfa70249d7827ad04630323e88 (127.10.71.131:35763)
I20260619 14:57:23.790781 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:57:23.791029 12976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:51379
W20260619 14:57:23.802970 13015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:23.801788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.10.71.190:35881)
0619 14:57:23.801913 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:23.801917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:23.802073 (+ 156us) server_negotiation.cc:407] Connection header received
0619 14:57:23.802158 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:23.802160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:23.802193 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:23.802256 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:23.802866 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37902: BlockingRecv error: recv got EOF from 127.0.0.1:37902 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:57:23.804069 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13038
I20260619 14:57:23.808892 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13172
I20260619 14:57:23.813861 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13306
I20260619 14:57:23.818969 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 12944
2026-06-19T14:57:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1910 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[13447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[13447](info): set up 2 sockets
Jun 19 14:57:23 dist-test-slave-3qdh krb5kdc[13447](info): commencing operation
krb5kdc: starting...
W20260619 14:57:24.784235 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:24 dist-test-slave-3qdh krb5kdc[13447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:24.894265 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:33439
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:24.988914 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:24.989120 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:24.989154 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:24.992843 13460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:24.992887 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:24.992905 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:24.992921 13460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:24.992936 13460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:24.997341 13460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:24.998049 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:24.998849 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:25.005880 13465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:25.005897 13466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:25.006083 13460 server_base.cc:1034] running on GCE node
W20260619 14:57:25.005880 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:25.006557 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:25.007472 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:25.008668 13460 hybrid_clock.cc:648] HybridClock initialized: now 1781881045008651 us; error 27 us; skew 500 ppm
Jun 19 14:57:25 dist-test-slave-3qdh krb5kdc[13447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881045, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:25.015919 13460 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:25.016724 13460 webserver.cc:460] Webserver started at http://127.10.71.190:41397/ using document root <none> and password file <none>
I20260619 14:57:25.017182 13460 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:25.017236 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:25.017439 13460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:25.019157 13460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "8383ded500c0446ab127900fd7d53d76"
format_stamp: "Formatted at 2026-06-19 14:57:25 on dist-test-slave-3qdh"
server_key: "79a6d4d31517873723f1707533f64547"
server_key_iv: "96e4002eef72fa34b63980f4c7377bd2"
server_key_version: "encryptionkey@0"
I20260619 14:57:25.019663 13460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "8383ded500c0446ab127900fd7d53d76"
format_stamp: "Formatted at 2026-06-19 14:57:25 on dist-test-slave-3qdh"
server_key: "79a6d4d31517873723f1707533f64547"
server_key_iv: "96e4002eef72fa34b63980f4c7377bd2"
server_key_version: "encryptionkey@0"
I20260619 14:57:25.022756 13460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:25.024824 13475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:25.025812 13460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:25.025936 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "8383ded500c0446ab127900fd7d53d76"
format_stamp: "Formatted at 2026-06-19 14:57:25 on dist-test-slave-3qdh"
server_key: "79a6d4d31517873723f1707533f64547"
server_key_iv: "96e4002eef72fa34b63980f4c7377bd2"
server_key_version: "encryptionkey@0"
I20260619 14:57:25.026027 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:25.036716 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:25.039628 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:25.039832 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:25.046226 13460 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:33439
I20260619 14:57:25.046482 13527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:33439 every 8 connection(s)
I20260619 14:57:25.047128 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:25.048522 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13460
I20260619 14:57:25.048650 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:25.048888 10526 external_mini_cluster.cc:1428] Setting key 538cfef93f3dad1d09db5a5f19dc6f6d
I20260619 14:57:25.049903 13528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:25.055877 13528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76: Bootstrap starting.
Jun 19 14:57:25 dist-test-slave-3qdh krb5kdc[13447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:25.057936 13528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:25.058748 13528 log.cc:824] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:25.060585 13528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76: No bootstrap required, opened a new log
I20260619 14:57:25.061926 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:25.050048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.10.71.190:33439)
0619 14:57:25.050417 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:25.050433 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:25.050467 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:57:25.051183 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:25.051202 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:25.051549 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:25.051906 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:25.052639 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:25.054024 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:25.054775 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:25.055400 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:25.055534 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:25.057753 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:25.057757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:25.057769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:25.057796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:25.059326 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:25.059682 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:25.059686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:25.059691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:25.059823 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:25.060059 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:25.060064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:25.060066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:25.060484 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:25.060666 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:25.060943 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:57:25.061173 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260619 14:57:25.064522 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:57:25.064440 13528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383ded500c0446ab127900fd7d53d76" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 33439 } }
I20260619 14:57:25.064668 13528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:25.064723 13528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8383ded500c0446ab127900fd7d53d76, State: Initialized, Role: FOLLOWER
I20260619 14:57:25.065129 13528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383ded500c0446ab127900fd7d53d76" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 33439 } }
I20260619 14:57:25.065233 13528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:25.065318 13528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:25.065395 13528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:25.066133 13528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383ded500c0446ab127900fd7d53d76" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 33439 } }
I20260619 14:57:25.066431 13528 leader_election.cc:302] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8383ded500c0446ab127900fd7d53d76; no voters:
I20260619 14:57:25.066685 13528 leader_election.cc:288] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:25.066797 13534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:25.067001 13534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [term 1 LEADER]: Becoming Leader. State: Replica: 8383ded500c0446ab127900fd7d53d76, State: Running, Role: LEADER
I20260619 14:57:25.067348 13534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383ded500c0446ab127900fd7d53d76" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 33439 } }
I20260619 14:57:25.067703 13528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:25.068473 13534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8383ded500c0446ab127900fd7d53d76. Latest consensus state: current_term: 1 leader_uuid: "8383ded500c0446ab127900fd7d53d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383ded500c0446ab127900fd7d53d76" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 33439 } } }
I20260619 14:57:25.068686 13534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:25.069154 13543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:25.070012 13535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8383ded500c0446ab127900fd7d53d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383ded500c0446ab127900fd7d53d76" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 33439 } } }
I20260619 14:57:25.070161 13535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:25.071947 13543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:25.076783 13543 catalog_manager.cc:1347] Generated new cluster ID: 09c43954465a48d28bba81689da5a8c0
I20260619 14:57:25.076839 13543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:25.085474 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:25.080221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.10.71.190:33439)
0619 14:57:25.080327 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:25.080331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:25.080716 (+ 385us) server_negotiation.cc:407] Connection header received
0619 14:57:25.080827 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:25.080830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:25.080869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:25.080961 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:25.081486 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:25.082188 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:25.083374 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:25.083714 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:25.083790 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:25.084614 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:25.084617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:25.084620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:25.085057 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:25.085084 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:25.085112 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:25.085192 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:25.085311 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:57:25.085355 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 14:57:25.086827 13543 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:25.087492 13543 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:25.094019 13543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8383ded500c0446ab127900fd7d53d76: Generated new TSK 0
I20260619 14:57:25.094501 13543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:57:25.099628 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13460
2026-06-19T14:57:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1277 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:25 dist-test-slave-3qdh krb5kdc[13561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:25 dist-test-slave-3qdh krb5kdc[13561](info): set up 2 sockets
Jun 19 14:57:25 dist-test-slave-3qdh krb5kdc[13561](info): commencing operation
krb5kdc: starting...
W20260619 14:57:26.044169 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:26.148797 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:36595
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:36595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:26.244638 13574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:26.244819 13574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:26.244853 13574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:26.248204 13574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:26.248245 13574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:26.248262 13574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:26.248276 13574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:26.248289 13574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:26.252792 13574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:36595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:36595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:26.253537 13574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:26.254281 13574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:26.261121 13582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.261143 13579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:26.261162 13574 server_base.cc:1034] running on GCE node
W20260619 14:57:26.261121 13580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:26.261839 13574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:26.262748 13574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:26.263983 13574 hybrid_clock.cc:648] HybridClock initialized: now 1781881046263986 us; error 45 us; skew 500 ppm
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:26.266327 13574 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:26.267061 13574 webserver.cc:460] Webserver started at http://127.10.71.190:44573/ using document root <none> and password file <none>
I20260619 14:57:26.267515 13574 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:26.267603 13574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:26.267807 13574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:26.269460 13574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "aaee568be3c44391b24275c61ad43fcb"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "f26b66c47f5140d3cfa0f6b33dbac9fc"
server_key_iv: "31de81df87a31609549a7be9edb74565"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.270257 13574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "aaee568be3c44391b24275c61ad43fcb"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "f26b66c47f5140d3cfa0f6b33dbac9fc"
server_key_iv: "31de81df87a31609549a7be9edb74565"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.273505 13574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:26.275458 13589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.276469 13574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:26.276583 13574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "aaee568be3c44391b24275c61ad43fcb"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "f26b66c47f5140d3cfa0f6b33dbac9fc"
server_key_iv: "31de81df87a31609549a7be9edb74565"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.276675 13574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:26.294615 13574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:26.296844 13574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:26.297010 13574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:26.303164 13574 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:36595
I20260619 14:57:26.303408 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:36595 every 8 connection(s)
I20260619 14:57:26.304024 13574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:26.306249 13642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:26.310497 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb: Bootstrap starting.
I20260619 14:57:26.312461 13642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:26.313115 13642 log.cc:824] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:26.313604 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13574
I20260619 14:57:26.313745 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:26.314018 10526 external_mini_cluster.cc:1428] Setting key d8414cee557b6af9e58adc991790e3d6
I20260619 14:57:26.315002 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb: No bootstrap required, opened a new log
I20260619 14:57:26.318044 13642 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaee568be3c44391b24275c61ad43fcb" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36595 } }
I20260619 14:57:26.318226 13642 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:26.318292 13642 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaee568be3c44391b24275c61ad43fcb, State: Initialized, Role: FOLLOWER
I20260619 14:57:26.318703 13642 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaee568be3c44391b24275c61ad43fcb" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36595 } }
I20260619 14:57:26.318845 13642 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:26.318907 13642 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:26.318991 13642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:26.319881 13642 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaee568be3c44391b24275c61ad43fcb" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36595 } }
I20260619 14:57:26.320222 13642 leader_election.cc:302] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaee568be3c44391b24275c61ad43fcb; no voters:
I20260619 14:57:26.320520 13642 leader_election.cc:288] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:26.320626 13647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:26.320871 13647 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [term 1 LEADER]: Becoming Leader. State: Replica: aaee568be3c44391b24275c61ad43fcb, State: Running, Role: LEADER
I20260619 14:57:26.321125 13647 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaee568be3c44391b24275c61ad43fcb" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36595 } }
I20260619 14:57:26.321738 13642 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:26.322420 13649 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaee568be3c44391b24275c61ad43fcb. Latest consensus state: current_term: 1 leader_uuid: "aaee568be3c44391b24275c61ad43fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaee568be3c44391b24275c61ad43fcb" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36595 } } }
I20260619 14:57:26.322557 13649 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:26.323143 13648 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaee568be3c44391b24275c61ad43fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaee568be3c44391b24275c61ad43fcb" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36595 } } }
I20260619 14:57:26.323254 13648 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:26.323869 13656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:26.326962 13656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:26.327745 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.314905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.10.71.190:36595)
0619 14:57:26.315288 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:26.315301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:26.315329 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:57:26.315828 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:26.315842 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:26.316086 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:26.316376 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:26.317166 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.318418 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.319541 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.320008 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.320142 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.322826 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:26.322832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:26.322846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:26.322874 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:26.324720 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.325149 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.325154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.325158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.325279 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.325514 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.325516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.325518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.325918 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:26.326090 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:26.326767 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 14:57:26.327005 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260619 14:57:26.331794 13656 catalog_manager.cc:1347] Generated new cluster ID: 729e6eefb04f424582f88fea081cb0ea
I20260619 14:57:26.331864 13656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:26.340695 13656 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:26.341751 13656 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:26.347833 13656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaee568be3c44391b24275c61ad43fcb: Generated new TSK 0
I20260619 14:57:26.348480 13656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:26.373371 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36595
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:26.469838 13668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:26.470032 13668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:26.470068 13668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:26.473763 13668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:26.473804 13668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:26.473887 13668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:26.478412 13668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:26.479202 13668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:26.480110 13668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:57:26.488345 13668 server_base.cc:1034] running on GCE node
W20260619 14:57:26.488257 13673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.488269 13674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.488276 13676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:26.488794 13668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:26.489302 13668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:26.490476 13668 hybrid_clock.cc:648] HybridClock initialized: now 1781881046490468 us; error 28 us; skew 500 ppm
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:26.492936 13668 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:26.493654 13668 webserver.cc:460] Webserver started at http://127.10.71.129:36701/ using document root <none> and password file <none>
I20260619 14:57:26.494138 13668 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:26.494218 13668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:26.494426 13668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:26.496011 13668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "2697a1e50d994b7f93e43f1bfc4c970d"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "7a2af7945fb67a5260826ceaf9a82466"
server_key_iv: "ad35028cbcb15da70604f38a54865bd3"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.496516 13668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "2697a1e50d994b7f93e43f1bfc4c970d"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "7a2af7945fb67a5260826ceaf9a82466"
server_key_iv: "ad35028cbcb15da70604f38a54865bd3"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.499480 13668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:57:26.501648 13683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.502743 13668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:26.502867 13668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "2697a1e50d994b7f93e43f1bfc4c970d"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "7a2af7945fb67a5260826ceaf9a82466"
server_key_iv: "ad35028cbcb15da70604f38a54865bd3"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.502965 13668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:26.529927 13668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:26.532686 13668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:26.532867 13668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:26.533352 13668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:26.534046 13668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:26.534107 13668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.534168 13668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:26.534214 13668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.541993 13668 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:44807
I20260619 14:57:26.542017 13796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:44807 every 8 connection(s)
I20260619 14:57:26.542696 13668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:26.548058 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13668
I20260619 14:57:26.548198 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:26.548521 10526 external_mini_cluster.cc:1428] Setting key 5000ddbe759c50784aa846c0d3820e4c
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:26.554929 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.544210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:49599 (local address 127.10.71.190:36595)
0619 14:57:26.544369 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:26.544372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:26.544959 (+ 587us) server_negotiation.cc:407] Connection header received
0619 14:57:26.546037 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:26.546039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:26.546086 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:26.546147 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:26.547018 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.547533 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.548774 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.549136 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.549242 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.551683 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:26.551686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:26.551688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:26.551708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:26.553053 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.553577 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.553583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.553585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.553664 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.553979 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.553983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.553985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.554142 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:26.554235 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:26.554630 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 14:57:26.554761 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:57:26.555511 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.544469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36595 (local address 127.10.71.129:49599)
0619 14:57:26.544829 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:26.544863 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:26.545883 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:26.546246 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:26.546251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:26.546537 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:26.546906 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:26.546917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.547625 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.547627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:26.548649 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:26.548658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.549279 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.549284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:26.549430 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.549963 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:26.549981 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:26.551534 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:26.553182 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:26.553189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:26.553203 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:26.553478 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:26.553773 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:26.553776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:26.553778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:26.553896 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:26.554234 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:26.554240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:26.554435 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 14:57:26.554591 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:57:26.554887 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260619 14:57:26.556555 13797 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36595
I20260619 14:57:26.556782 13797 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:26.557279 13797 heartbeater.cc:511] Master 127.10.71.190:36595 requested a full tablet report, sending...
I20260619 14:57:26.558516 13606 ts_manager.cc:194] Registered new tserver with Master: 2697a1e50d994b7f93e43f1bfc4c970d (127.10.71.129:44807)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:26.559626 13606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:49599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:26.572372 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36595
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:26.667932 13802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:26.668123 13802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:26.668159 13802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:26.671705 13802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:26.671747 13802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:26.671799 13802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:26.676450 13802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:26.677147 13802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:26.677912 13802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:57:26.685962 13802 server_base.cc:1034] running on GCE node
W20260619 14:57:26.685978 13808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.685973 13807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.686128 13810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:26.686573 13802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:26.687115 13802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:26.688285 13802 hybrid_clock.cc:648] HybridClock initialized: now 1781881046688262 us; error 43 us; skew 500 ppm
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:26.690675 13802 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:26.691351 13802 webserver.cc:460] Webserver started at http://127.10.71.130:34695/ using document root <none> and password file <none>
I20260619 14:57:26.691797 13802 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:26.691848 13802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:26.691980 13802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:26.693761 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "f69e221cc7d44c3d93238e8cd3d42e56"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "f352f92068f5a5f8f322fad7ec6ddba9"
server_key_iv: "5a5e2d626df8ebe80deca8dcdbcf1d3d"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.694183 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "f69e221cc7d44c3d93238e8cd3d42e56"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "f352f92068f5a5f8f322fad7ec6ddba9"
server_key_iv: "5a5e2d626df8ebe80deca8dcdbcf1d3d"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.697247 13802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:57:26.699215 13817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.700177 13802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:57:26.700261 13802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "f69e221cc7d44c3d93238e8cd3d42e56"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "f352f92068f5a5f8f322fad7ec6ddba9"
server_key_iv: "5a5e2d626df8ebe80deca8dcdbcf1d3d"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.700382 13802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:26.718365 13802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:26.721472 13802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:26.721609 13802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:26.722025 13802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:26.722855 13802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:26.722904 13802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.722944 13802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:26.722968 13802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.731974 13802 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:46355
I20260619 14:57:26.731994 13930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:46355 every 8 connection(s)
I20260619 14:57:26.732888 13802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:26.737187 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13802
I20260619 14:57:26.737279 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:26.737592 10526 external_mini_cluster.cc:1428] Setting key d978d30a42df8fd2d908d0fdc647f183
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:26.745328 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.734472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:49023 (local address 127.10.71.190:36595)
0619 14:57:26.734622 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:26.734625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:26.735360 (+ 735us) server_negotiation.cc:407] Connection header received
0619 14:57:26.736486 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:26.736489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:26.736541 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:26.736633 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:26.737666 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.738141 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.739115 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.739318 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.739415 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.741851 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:26.741857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:26.741863 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:26.741893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:26.743489 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.743968 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.743973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.743975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.744051 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.744390 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.744393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.744395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.744556 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:26.744671 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:26.745071 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:57:26.745185 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:57:26.746078 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.734803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36595 (local address 127.10.71.130:49023)
0619 14:57:26.735222 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:26.735256 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:26.736282 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:26.736781 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:26.736787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:26.737051 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:26.737528 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:26.737541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.738255 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.738258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:26.738978 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:26.738985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.739444 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.739448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:26.739598 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.740088 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:26.740102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:26.741698 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:26.743598 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:26.743603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:26.743612 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:26.743862 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:26.744151 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:26.744154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:26.744156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:26.744274 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:26.744671 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:26.744677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:26.744863 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 14:57:26.745016 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:57:26.745286 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:57:26.747126 13931 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36595
I20260619 14:57:26.747339 13931 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:26.747754 13931 heartbeater.cc:511] Master 127.10.71.190:36595 requested a full tablet report, sending...
I20260619 14:57:26.748719 13606 ts_manager.cc:194] Registered new tserver with Master: f69e221cc7d44c3d93238e8cd3d42e56 (127.10.71.130:46355)
I20260619 14:57:26.749290 13606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:49023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:26.761288 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36595
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:26.863157 13936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:26.863350 13936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:26.863389 13936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:26.866883 13936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:26.866925 13936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:26.866972 13936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:26.871610 13936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:26.872377 13936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:26.873198 13936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:26.881789 13944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.881800 13942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:26.881780 13941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:26.881979 13936 server_base.cc:1034] running on GCE node
I20260619 14:57:26.882426 13936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:26.883042 13936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:26.884191 13936 hybrid_clock.cc:648] HybridClock initialized: now 1781881046884175 us; error 31 us; skew 500 ppm
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:26.886695 13936 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:26.887408 13936 webserver.cc:460] Webserver started at http://127.10.71.131:33699/ using document root <none> and password file <none>
I20260619 14:57:26.887882 13936 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:26.887934 13936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:26.888108 13936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:26.889825 13936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "3b9ae24760c24608bc4f2e7d916c9457"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "46257268e475e5749283f9f496659310"
server_key_iv: "484781b55feb6efc9cd58c766922bce6"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.890311 13936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "3b9ae24760c24608bc4f2e7d916c9457"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "46257268e475e5749283f9f496659310"
server_key_iv: "484781b55feb6efc9cd58c766922bce6"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.893427 13936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:57:26.895525 13951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.896510 13936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:26.896646 13936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "3b9ae24760c24608bc4f2e7d916c9457"
format_stamp: "Formatted at 2026-06-19 14:57:26 on dist-test-slave-3qdh"
server_key: "46257268e475e5749283f9f496659310"
server_key_iv: "484781b55feb6efc9cd58c766922bce6"
server_key_version: "encryptionkey@0"
I20260619 14:57:26.896737 13936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:26.910486 13936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:26.912886 13936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:26.913067 13936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:26.913542 13936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:26.914203 13936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:26.914268 13936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.914325 13936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:26.914371 13936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:26.922257 13936 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:44705
I20260619 14:57:26.922282 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:44705 every 8 connection(s)
I20260619 14:57:26.923085 13936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:26.926036 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 13936
I20260619 14:57:26.926193 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:26.926482 10526 external_mini_cluster.cc:1428] Setting key 6c0f5842ce5fcf5eb8a9d3debc4fb93a
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:26.935652 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.924645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:35111 (local address 127.10.71.190:36595)
0619 14:57:26.924760 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:26.924763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:26.925414 (+ 651us) server_negotiation.cc:407] Connection header received
0619 14:57:26.926954 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:26.926958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:26.927027 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:26.927126 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:26.928244 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.928869 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.929836 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.930074 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.930155 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.932157 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:26.932160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:26.932163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:26.932187 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:26.933716 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.934257 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.934263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.934264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.934350 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.934681 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.934684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.934686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.934855 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:26.934955 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:26.935395 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 14:57:26.935497 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:26.936234 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.924927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36595 (local address 127.10.71.131:35111)
0619 14:57:26.925294 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:26.925327 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:26.926748 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:26.927333 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:26.927340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:26.927632 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:26.928117 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:26.928129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.928978 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.928980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:26.929730 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:26.929739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.930191 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.930193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:26.930312 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.930753 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:26.930767 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:26.932012 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:26.933866 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:26.933871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:26.933883 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:26.934157 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:26.934443 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:26.934448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:26.934450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:26.934582 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:26.934966 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:26.934973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:26.935172 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 14:57:26.935341 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:57:26.935599 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260619 14:57:26.937232 14065 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36595
I20260619 14:57:26.937394 14065 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:26.937732 14065 heartbeater.cc:511] Master 127.10.71.190:36595 requested a full tablet report, sending...
I20260619 14:57:26.938516 13606 ts_manager.cc:194] Registered new tserver with Master: 3b9ae24760c24608bc4f2e7d916c9457 (127.10.71.131:44705)
I20260619 14:57:26.939097 13606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:35111
I20260619 14:57:26.940125 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:57:26 dist-test-slave-3qdh krb5kdc[13561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881046, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:26.962695 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:26.955434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.10.71.190:36595)
0619 14:57:26.955548 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:26.955552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:26.955717 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:57:26.955818 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:26.955820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:26.955848 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:26.955902 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:26.956566 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.956962 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.957642 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:26.957804 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:26.957862 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:26.959988 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:26.959990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:26.959992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:26.960010 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:26.961194 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.961560 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.961563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.961565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.961631 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:26.961885 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:26.961888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:26.961890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:26.962057 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:26.962183 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:26.962403 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 14:57:26.962521 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:57:26.963002 13606 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34248
I20260619 14:57:26.963879 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13668
I20260619 14:57:26.969091 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13802
I20260619 14:57:26.973868 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13936
I20260619 14:57:26.978616 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 13574
2026-06-19T14:57:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1881 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:27 dist-test-slave-3qdh krb5kdc[14077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:27 dist-test-slave-3qdh krb5kdc[14077](info): set up 2 sockets
Jun 19 14:57:27 dist-test-slave-3qdh krb5kdc[14077](info): commencing operation
krb5kdc: starting...
W20260619 14:57:27.976351 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:28.089095 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40253
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:40253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:28.193602 14090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:28.193830 14090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:28.193867 14090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:28.197777 14090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:28.197826 14090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:28.197845 14090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:28.197861 14090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:28.197903 14090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:28.202888 14090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:40253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:28.203857 14090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:28.204854 14090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:28.212827 14095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.212839 14096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:28.212870 14090 server_base.cc:1034] running on GCE node
W20260619 14:57:28.212922 14098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:28.213589 14090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:28.214543 14090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:28.215735 14090 hybrid_clock.cc:648] HybridClock initialized: now 1781881048215728 us; error 27 us; skew 500 ppm
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:28.218385 14090 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:28.219137 14090 webserver.cc:460] Webserver started at http://127.10.71.190:36349/ using document root <none> and password file <none>
I20260619 14:57:28.219623 14090 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:28.219679 14090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:28.219844 14090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:28.221763 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "885971642893409380df03a8cbaa5884"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "a0cd72ccad3bf0e98817667f785787ce"
server_key_iv: "3d1fda0b57a2eedccebb4bb6b37a6812"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.222244 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "885971642893409380df03a8cbaa5884"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "a0cd72ccad3bf0e98817667f785787ce"
server_key_iv: "3d1fda0b57a2eedccebb4bb6b37a6812"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.225668 14090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:57:28.232928 14105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.234206 14090 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260619 14:57:28.234328 14090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "885971642893409380df03a8cbaa5884"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "a0cd72ccad3bf0e98817667f785787ce"
server_key_iv: "3d1fda0b57a2eedccebb4bb6b37a6812"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.234431 14090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:28.245492 14090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:28.248104 14090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:28.248291 14090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:28.255347 14090 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:40253
I20260619 14:57:28.255358 14157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:40253 every 8 connection(s)
I20260619 14:57:28.256278 14090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:28.258875 14158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:28.263809 14158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884: Bootstrap starting.
I20260619 14:57:28.264564 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14090
I20260619 14:57:28.264693 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:28.264974 10526 external_mini_cluster.cc:1428] Setting key 8ae758e68711dac3a23d4c55527dade4
I20260619 14:57:28.266379 14158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:28.267057 14158 log.cc:824] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:28.269138 14158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884: No bootstrap required, opened a new log
I20260619 14:57:28.273334 14158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885971642893409380df03a8cbaa5884" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40253 } }
I20260619 14:57:28.273624 14158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:28.273685 14158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885971642893409380df03a8cbaa5884, State: Initialized, Role: FOLLOWER
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:28.274134 14158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885971642893409380df03a8cbaa5884" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40253 } }
I20260619 14:57:28.274288 14158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:28.274349 14158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:28.274441 14158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:28.275244 14158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885971642893409380df03a8cbaa5884" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40253 } }
I20260619 14:57:28.275607 14158 leader_election.cc:302] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885971642893409380df03a8cbaa5884; no voters:
I20260619 14:57:28.275921 14158 leader_election.cc:288] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:28.276096 14163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:28.276429 14163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [term 1 LEADER]: Becoming Leader. State: Replica: 885971642893409380df03a8cbaa5884, State: Running, Role: LEADER
I20260619 14:57:28.276827 14163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885971642893409380df03a8cbaa5884" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40253 } }
I20260619 14:57:28.277244 14158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:28.278460 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885971642893409380df03a8cbaa5884. Latest consensus state: current_term: 1 leader_uuid: "885971642893409380df03a8cbaa5884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885971642893409380df03a8cbaa5884" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40253 } } }
I20260619 14:57:28.278587 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:28.278949 14164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "885971642893409380df03a8cbaa5884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885971642893409380df03a8cbaa5884" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40253 } } }
I20260619 14:57:28.279071 14164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:28.280861 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.266099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.10.71.190:40253)
0619 14:57:28.266517 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:28.266549 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:28.266585 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:57:28.267199 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:28.267221 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:28.267635 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:28.267969 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:28.268874 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.269874 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.271048 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.271572 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.271694 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.274843 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:28.274848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:28.274859 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:28.274886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:28.276977 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.277507 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.277514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.277519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.277663 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.278011 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.278016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.278021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.278546 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:28.278819 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:28.279595 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 14:57:28.279897 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":132,"threads_started":1}
W20260619 14:57:28.281741 14178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:57:28.281816 14178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:57:28.282776 14173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:28.283639 14173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:28.288865 14173 catalog_manager.cc:1347] Generated new cluster ID: 1ea600d3cc3249f0bb5caeb4d55b0a13
I20260619 14:57:28.288944 14173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:28.305555 14173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:28.306422 14173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:28.316879 14173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 885971642893409380df03a8cbaa5884: Generated new TSK 0
I20260619 14:57:28.317579 14173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:28.351449 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40253
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:28.457808 14184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:28.458081 14184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:28.458168 14184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:28.462121 14184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:28.462204 14184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:28.462303 14184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:28.467273 14184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:28.468253 14184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:28.469218 14184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:28.478269 14189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.478215 14190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.478217 14192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:28.478466 14184 server_base.cc:1034] running on GCE node
I20260619 14:57:28.478958 14184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:28.479594 14184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:28.480751 14184 hybrid_clock.cc:648] HybridClock initialized: now 1781881048480731 us; error 33 us; skew 500 ppm
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:28.483429 14184 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:28.484201 14184 webserver.cc:460] Webserver started at http://127.10.71.129:45083/ using document root <none> and password file <none>
I20260619 14:57:28.484776 14184 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:28.484870 14184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:28.485107 14184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:28.486943 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "c0cf3399570349cf871351d01b5194b3"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "eeb3d06837e4e10d851bc5ce754b406f"
server_key_iv: "d54810dd370a7b6c613efe13001471fa"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.487512 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "c0cf3399570349cf871351d01b5194b3"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "eeb3d06837e4e10d851bc5ce754b406f"
server_key_iv: "d54810dd370a7b6c613efe13001471fa"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.490931 14184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:57:28.493542 14199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.494635 14184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:28.494774 14184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "c0cf3399570349cf871351d01b5194b3"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "eeb3d06837e4e10d851bc5ce754b406f"
server_key_iv: "d54810dd370a7b6c613efe13001471fa"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.494879 14184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:28.518267 14184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:28.520882 14184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:28.521086 14184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:28.521611 14184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:28.522351 14184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:28.522419 14184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.522493 14184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:28.522547 14184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.531387 14184 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:38749
I20260619 14:57:28.531430 14312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:38749 every 8 connection(s)
I20260619 14:57:28.532176 14184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:28.537011 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14184
I20260619 14:57:28.537154 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:28.537453 10526 external_mini_cluster.cc:1428] Setting key c499fa421dcecb27af31efe45f616a45
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:28.546994 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.533851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:37677 (local address 127.10.71.190:40253)
0619 14:57:28.533994 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:28.533997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:28.534709 (+ 712us) server_negotiation.cc:407] Connection header received
0619 14:57:28.536006 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:28.536010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:28.536066 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:28.536174 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:28.537411 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.538146 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.539804 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.540036 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.540137 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.542998 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:28.543004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:28.543008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:28.543037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:28.544855 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.545338 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.545344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.545348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.545452 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.545777 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.545781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.545783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.546044 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:28.546166 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:28.546636 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 14:57:28.546771 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:28.547554 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.534162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40253 (local address 127.10.71.129:37677)
0619 14:57:28.534572 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:28.534612 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:28.535813 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:28.536372 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:28.536379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:28.536698 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:28.537274 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:28.537289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.538287 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.538290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:28.539653 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:28.539667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.540196 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.540203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:28.540464 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.541094 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:28.541109 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:28.542821 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:28.544977 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:28.544982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:28.544992 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:28.545229 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:28.545553 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:28.545555 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:28.545556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:28.545678 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:28.546180 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:28.546185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:28.546379 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 14:57:28.546532 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:57:28.546748 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260619 14:57:28.548724 14313 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40253
I20260619 14:57:28.548965 14313 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:28.549467 14313 heartbeater.cc:511] Master 127.10.71.190:40253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:28.551007 14122 ts_manager.cc:194] Registered new tserver with Master: c0cf3399570349cf871351d01b5194b3 (127.10.71.129:38749)
I20260619 14:57:28.552130 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:37677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:28.564960 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40253
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:28.666893 14318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:28.667141 14318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:28.667214 14318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:28.671072 14318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:28.671150 14318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:28.671234 14318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:28.676131 14318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:28.676993 14318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:28.677883 14318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:28.686626 14324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.686666 14323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.686647 14326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:28.686820 14318 server_base.cc:1034] running on GCE node
I20260619 14:57:28.687193 14318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:28.687741 14318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:28.688932 14318 hybrid_clock.cc:648] HybridClock initialized: now 1781881048688911 us; error 45 us; skew 500 ppm
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:28.691502 14318 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:28.692286 14318 webserver.cc:460] Webserver started at http://127.10.71.130:39265/ using document root <none> and password file <none>
I20260619 14:57:28.693038 14318 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:28.693120 14318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:28.693308 14318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:28.695150 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "157f9e0670de4e6c822a143ca9fdd0d2"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "bb68de4a059d0e933652c4928c19a239"
server_key_iv: "12794e97d3f21d53fdb487884608d9d5"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.695673 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "157f9e0670de4e6c822a143ca9fdd0d2"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "bb68de4a059d0e933652c4928c19a239"
server_key_iv: "12794e97d3f21d53fdb487884608d9d5"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.699137 14318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 14:57:28.701329 14333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.702370 14318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:28.702493 14318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "157f9e0670de4e6c822a143ca9fdd0d2"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "bb68de4a059d0e933652c4928c19a239"
server_key_iv: "12794e97d3f21d53fdb487884608d9d5"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.702589 14318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:28.728591 14318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:28.731069 14318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:28.731240 14318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:28.731769 14318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:28.732607 14318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:28.732671 14318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.732753 14318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:28.732810 14318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.741765 14318 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:37929
I20260619 14:57:28.741793 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:37929 every 8 connection(s)
I20260619 14:57:28.742511 14318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:28.750314 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14318
I20260619 14:57:28.750458 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:28.750720 10526 external_mini_cluster.cc:1428] Setting key 9142f4602fb724b91c78eeb8a6338813
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:28.754889 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.744090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:42079 (local address 127.10.71.190:40253)
0619 14:57:28.744249 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:28.744254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:28.744984 (+ 730us) server_negotiation.cc:407] Connection header received
0619 14:57:28.746173 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:28.746176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:28.746220 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:28.746291 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:28.747268 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.747766 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.748816 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.749016 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.749106 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.751466 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:28.751469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:28.751470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:28.751491 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:28.752846 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.753437 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.753440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.753441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.753541 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.753873 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.753878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.753880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.754056 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:28.754156 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:28.754630 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 14:57:28.754733 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:57:28.755543 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.744460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40253 (local address 127.10.71.130:42079)
0619 14:57:28.744848 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:28.744883 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:28.745999 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:28.746427 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:28.746433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:28.746734 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:28.747147 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:28.747158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.747899 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.747901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:28.748720 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:28.748728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.749149 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.749151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:28.749273 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.749749 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:28.749763 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:28.751315 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:28.752961 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:28.752969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:28.752985 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:28.753343 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:28.753641 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:28.753645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:28.753647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:28.753776 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:28.754155 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:28.754161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:28.754364 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 14:57:28.754577 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 14:57:28.754865 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260619 14:57:28.756687 14447 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40253
I20260619 14:57:28.756894 14447 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:28.757287 14447 heartbeater.cc:511] Master 127.10.71.190:40253 requested a full tablet report, sending...
I20260619 14:57:28.758234 14122 ts_manager.cc:194] Registered new tserver with Master: 157f9e0670de4e6c822a143ca9fdd0d2 (127.10.71.130:37929)
I20260619 14:57:28.758771 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:42079
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:28.775308 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40253
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:28.878124 14452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:28.878314 14452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:28.878405 14452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:28.882647 14452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:28.882718 14452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:28.882776 14452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:28.887655 14452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:28.888451 14452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:28.889238 14452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:28.897469 14458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.897468 14460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:28.897519 14457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:28.897670 14452 server_base.cc:1034] running on GCE node
I20260619 14:57:28.898120 14452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:28.898625 14452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:28.899791 14452 hybrid_clock.cc:648] HybridClock initialized: now 1781881048899771 us; error 59 us; skew 500 ppm
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:28.902175 14452 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:28.902937 14452 webserver.cc:460] Webserver started at http://127.10.71.131:35835/ using document root <none> and password file <none>
I20260619 14:57:28.903419 14452 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:28.903505 14452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:28.903709 14452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:28.905405 14452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "2cef9755734f4aafb3b31917f6f6f8f3"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "386fbf42272fb9c796ed521f0bb39255"
server_key_iv: "310b434f3602d38c8a7152b27de859c2"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.905913 14452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "2cef9755734f4aafb3b31917f6f6f8f3"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "386fbf42272fb9c796ed521f0bb39255"
server_key_iv: "310b434f3602d38c8a7152b27de859c2"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.909005 14452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:28.910890 14467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.911923 14452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:28.912079 14452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "2cef9755734f4aafb3b31917f6f6f8f3"
format_stamp: "Formatted at 2026-06-19 14:57:28 on dist-test-slave-3qdh"
server_key: "386fbf42272fb9c796ed521f0bb39255"
server_key_iv: "310b434f3602d38c8a7152b27de859c2"
server_key_version: "encryptionkey@0"
I20260619 14:57:28.912175 14452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:28.929260 14452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:28.931897 14452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:28.932053 14452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:28.932580 14452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:28.933318 14452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:28.933384 14452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.933449 14452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:28.933499 14452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:28.941438 14452 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:46041
I20260619 14:57:28.941463 14580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:46041 every 8 connection(s)
I20260619 14:57:28.942152 14452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:28.949862 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14452
I20260619 14:57:28.949980 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:28.950278 10526 external_mini_cluster.cc:1428] Setting key 124595680d0593edbcc778352199b87f
I20260619 14:57:28.953922 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.943604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:44541 (local address 127.10.71.190:40253)
0619 14:57:28.943726 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:28.943729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:28.944417 (+ 688us) server_negotiation.cc:407] Connection header received
0619 14:57:28.945644 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:28.945647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:28.945706 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:28.945811 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:28.946809 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.947336 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.948344 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.948555 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.948659 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.950717 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:28.950721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:28.950723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:28.950752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:28.952228 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.952716 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.952718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.952719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.952803 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.953093 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.953095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.953097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.953253 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:28.953323 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:28.953710 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 14:57:28.953794 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:57:28.954557 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.943901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40253 (local address 127.10.71.131:44541)
0619 14:57:28.944275 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:28.944322 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:28.945432 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:28.945972 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:28.945978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:28.946275 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:28.946663 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:28.946674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.947448 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.947451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:28.948216 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:28.948224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.948685 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.948689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:28.948836 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.949219 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:28.949232 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:28.950576 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:28.952392 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:28.952396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:28.952405 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:28.952630 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:28.952883 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:28.952886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:28.952887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:28.953001 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:28.953335 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:28.953339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:28.953526 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 14:57:28.953663 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:57:28.953872 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260619 14:57:28.955459 14581 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40253
I20260619 14:57:28.955662 14581 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:28.956045 14581 heartbeater.cc:511] Master 127.10.71.190:40253 requested a full tablet report, sending...
I20260619 14:57:28.956872 14122 ts_manager.cc:194] Registered new tserver with Master: 2cef9755734f4aafb3b31917f6f6f8f3 (127.10.71.131:46041)
I20260619 14:57:28.957326 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:44541
I20260619 14:57:28.963804 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:28.987922 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.978963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.10.71.129:38749)
0619 14:57:28.979289 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:28.979305 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:28.979333 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:57:28.979392 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:28.979404 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:28.979568 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:28.979717 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:28.980486 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.981216 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.982050 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.982425 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.982559 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.984982 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:28.984989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:28.984999 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:28.985019 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:28.986291 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.986722 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.986732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.986733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.986805 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.987054 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.987058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.987060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.987267 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:28.987455 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:28.987627 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 14:57:28.987760 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
Jun 19 14:57:28 dist-test-slave-3qdh krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:28.997686 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:28.990649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.10.71.190:40253)
0619 14:57:28.990784 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:28.990787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:28.990861 (+ 74us) server_negotiation.cc:407] Connection header received
0619 14:57:28.991001 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:28.991004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:28.991051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:28.991133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:28.991649 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.992092 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.992871 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:28.993103 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:28.993211 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:28.995206 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:28.995210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:28.995212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:28.995240 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:28.996283 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.996660 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.996663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.996665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.996733 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:28.996969 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:28.996972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:28.996973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:28.997149 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:28.997269 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:28.997453 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:57:28.997550 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:57:28.998015 14122 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56818
I20260619 14:57:28.998569 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14184
I20260619 14:57:29.003830 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14318
I20260619 14:57:29.008329 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14452
I20260619 14:57:29.013162 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14090
2026-06-19T14:57:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2034 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:29 dist-test-slave-3qdh krb5kdc[14593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:29 dist-test-slave-3qdh krb5kdc[14593](info): set up 2 sockets
Jun 19 14:57:29 dist-test-slave-3qdh krb5kdc[14593](info): commencing operation
krb5kdc: starting...
W20260619 14:57:29.964195 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:30.068917 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:44717
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:30.163727 14606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:30.164002 14606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:30.164058 14606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:30.167497 14606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:30.167538 14606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:30.167554 14606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:30.167573 14606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:30.167586 14606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:30.171977 14606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:30.172703 14606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:30.173486 14606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:30.180150 14614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:30.180124 14611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.180377 14606 server_base.cc:1034] running on GCE node
W20260619 14:57:30.180135 14612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.180848 14606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:30.181722 14606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:30.182906 14606 hybrid_clock.cc:648] HybridClock initialized: now 1781881050182877 us; error 40 us; skew 500 ppm
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:30.190936 14606 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:30.191489 14606 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:30.191582 14606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:30.191766 14606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:30.193528 14606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "299ecf53d87b447f8a1031619d51b880"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "031802068358bb6e09c041a046e8e4d6"
server_key_iv: "534de4ef0f0a3a3a4b0fa7bac0e0e92a"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.194054 14606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "299ecf53d87b447f8a1031619d51b880"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "031802068358bb6e09c041a046e8e4d6"
server_key_iv: "534de4ef0f0a3a3a4b0fa7bac0e0e92a"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.196960 14606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:30.198784 14620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.199743 14606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:30.199859 14606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "299ecf53d87b447f8a1031619d51b880"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "031802068358bb6e09c041a046e8e4d6"
server_key_iv: "534de4ef0f0a3a3a4b0fa7bac0e0e92a"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.199946 14606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:30.210199 14606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:30.212218 14606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:30.212401 14606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:30.218075 14606 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:44717
I20260619 14:57:30.218276 14672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:44717 every 8 connection(s)
I20260619 14:57:30.218575 14606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:30.220741 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:30.223151 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14606
I20260619 14:57:30.223271 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:30.223524 10526 external_mini_cluster.cc:1428] Setting key 2932282ca972914423ea6b8a6cc2cefc
I20260619 14:57:30.225814 14673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: Bootstrap starting.
I20260619 14:57:30.228015 14673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:30.228776 14673 log.cc:824] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:30.230510 14673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: No bootstrap required, opened a new log
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:30.233547 14673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299ecf53d87b447f8a1031619d51b880" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44717 } }
I20260619 14:57:30.233795 14673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:30.233843 14673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299ecf53d87b447f8a1031619d51b880, State: Initialized, Role: FOLLOWER
I20260619 14:57:30.234287 14673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299ecf53d87b447f8a1031619d51b880" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44717 } }
I20260619 14:57:30.234452 14673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:30.234548 14673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:30.234648 14673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:30.235540 14673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299ecf53d87b447f8a1031619d51b880" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44717 } }
I20260619 14:57:30.235867 14673 leader_election.cc:302] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 299ecf53d87b447f8a1031619d51b880; no voters:
I20260619 14:57:30.236119 14673 leader_election.cc:288] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:30.236341 14678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:30.236526 14678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [term 1 LEADER]: Becoming Leader. State: Replica: 299ecf53d87b447f8a1031619d51b880, State: Running, Role: LEADER
I20260619 14:57:30.236850 14678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299ecf53d87b447f8a1031619d51b880" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44717 } }
I20260619 14:57:30.237319 14673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:30.237694 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.224743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.10.71.190:44717)
0619 14:57:30.225107 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.225123 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.225159 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:57:30.225867 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.225887 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.226222 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.226508 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:30.227076 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.228371 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.229221 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.229684 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.229783 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.231949 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:30.231953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:30.231962 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:30.231984 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:30.234682 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.235266 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.235272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.235277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.235407 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.235739 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.235742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.235743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.236150 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:30.236363 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.236671 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.236935 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260619 14:57:30.237917 14678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 299ecf53d87b447f8a1031619d51b880. Latest consensus state: current_term: 1 leader_uuid: "299ecf53d87b447f8a1031619d51b880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299ecf53d87b447f8a1031619d51b880" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44717 } } }
I20260619 14:57:30.238071 14678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:30.238621 14680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "299ecf53d87b447f8a1031619d51b880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299ecf53d87b447f8a1031619d51b880" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44717 } } }
I20260619 14:57:30.238768 14680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880 [sys.catalog]: This master's current role is: LEADER
W20260619 14:57:30.242249 14693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:57:30.242307 14693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:57:30.242380 14685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:30.243013 14685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:30.247874 14685 catalog_manager.cc:1347] Generated new cluster ID: 4e5a93cc2a704702a5da480f44ef2b2b
I20260619 14:57:30.247937 14685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:30.270123 14685 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:30.271077 14685 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:30.276139 14685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: Generated new TSK 0
I20260619 14:57:30.276671 14685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:30.306505 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:44717
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:30.405035 14699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:30.405227 14699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:30.405265 14699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:30.409046 14699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:30.409089 14699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:30.409142 14699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:30.413810 14699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:44717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:30.414532 14699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:30.415315 14699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:30.423398 14704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:30.423434 14707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.423434 14699 server_base.cc:1034] running on GCE node
W20260619 14:57:30.423596 14705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.423913 14699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:30.424536 14699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:30.425734 14699 hybrid_clock.cc:648] HybridClock initialized: now 1781881050425713 us; error 43 us; skew 500 ppm
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:30.428066 14699 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:30.428738 14699 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:30.428826 14699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:30.429029 14699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:30.430716 14699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "a78f9565e5a5465a8970a74e9a780ae5"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "70cdbc0ba6d7006276d925c196740eff"
server_key_iv: "0f983db112d373e40cdd21962044fd1f"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.431195 14699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "a78f9565e5a5465a8970a74e9a780ae5"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "70cdbc0ba6d7006276d925c196740eff"
server_key_iv: "0f983db112d373e40cdd21962044fd1f"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.434482 14699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:30.436513 14713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.437583 14699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:30.437716 14699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "a78f9565e5a5465a8970a74e9a780ae5"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "70cdbc0ba6d7006276d925c196740eff"
server_key_iv: "0f983db112d373e40cdd21962044fd1f"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.437839 14699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:30.454545 14699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:30.456914 14699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:30.457083 14699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:30.457549 14699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:30.458341 14699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:30.458384 14699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.458446 14699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:30.458477 14699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.466584 14699 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:43359
I20260619 14:57:30.466605 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:43359 every 8 connection(s)
I20260619 14:57:30.467064 14699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:30.471041 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14699
I20260619 14:57:30.471175 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:30.471453 10526 external_mini_cluster.cc:1428] Setting key 5ae796218cfd2a485cf30febbc5e24d5
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:30.479944 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.468551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:50279 (local address 127.10.71.190:44717)
0619 14:57:30.468676 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.468679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.469431 (+ 752us) server_negotiation.cc:407] Connection header received
0619 14:57:30.470596 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.470600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.470670 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.470767 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:30.471956 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.472645 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.473739 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.474180 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.474321 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.476723 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:30.476726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:30.476728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:30.476749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:30.478121 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.478622 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.478625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.478627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.478700 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.478998 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.479002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.479004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.479160 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:30.479284 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.479695 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.479800 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:57:30.480576 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.468910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:44717 (local address 127.10.71.129:50279)
0619 14:57:30.469298 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.469334 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.470383 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.470910 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.470916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.471336 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:30.471788 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.471805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.472763 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.472768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.473614 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.473623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.474365 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.474373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.474586 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.475099 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:30.475120 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:30.476553 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:30.478263 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:30.478268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:30.478280 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:30.478509 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:30.478785 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:30.478789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:30.478791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:30.478911 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:30.479281 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:30.479287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:30.479492 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.479646 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.479894 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":112,"threads_started":1}
I20260619 14:57:30.481614 14827 heartbeater.cc:347] Connected to a master server at 127.10.71.190:44717
I20260619 14:57:30.481793 14827 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:30.482167 14827 heartbeater.cc:511] Master 127.10.71.190:44717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:30.483474 14637 ts_manager.cc:194] Registered new tserver with Master: a78f9565e5a5465a8970a74e9a780ae5 (127.10.71.129:43359)
I20260619 14:57:30.485116 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:50279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:30.496511 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:44717
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:30.592077 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:30.592348 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:30.592406 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:30.595975 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:30.596014 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:30.596064 14832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:30.601006 14832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:44717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:30.601722 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:30.602566 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:30.610610 14837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:30.610574 14840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:30.610577 14838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.610776 14832 server_base.cc:1034] running on GCE node
I20260619 14:57:30.611167 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:30.611686 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:30.612877 14832 hybrid_clock.cc:648] HybridClock initialized: now 1781881050612865 us; error 51 us; skew 500 ppm
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:30.615331 14832 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:30.615953 14832 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:30.616032 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:30.616238 14832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:30.617915 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "277f0ed7eb5046179e424d286c85056e"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "33e570f67a534ec7516380ef98c2aa86"
server_key_iv: "87ad8f98c509e2f9e0fd0f67e4eac87a"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.618376 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "277f0ed7eb5046179e424d286c85056e"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "33e570f67a534ec7516380ef98c2aa86"
server_key_iv: "87ad8f98c509e2f9e0fd0f67e4eac87a"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.621243 14832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:30.623196 14846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.624385 14832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:30.624496 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "277f0ed7eb5046179e424d286c85056e"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "33e570f67a534ec7516380ef98c2aa86"
server_key_iv: "87ad8f98c509e2f9e0fd0f67e4eac87a"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.624606 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:30.646044 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:30.648267 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:30.648478 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:30.648948 14832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:30.649632 14832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:30.649696 14832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.649756 14832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:30.649788 14832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.657322 14832 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:44693
I20260619 14:57:30.657361 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:44693 every 8 connection(s)
I20260619 14:57:30.657799 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:30.661181 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14832
I20260619 14:57:30.661302 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:30.661548 10526 external_mini_cluster.cc:1428] Setting key 19cf5adc507964ed7b49aac5b2e880ac
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:57:30.669714 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.659293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:47967 (local address 127.10.71.190:44717)
0619 14:57:30.659410 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.659414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.660050 (+ 636us) server_negotiation.cc:407] Connection header received
0619 14:57:30.661169 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.661174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.661225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.661311 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:30.662448 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.662966 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.664065 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.664290 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.664423 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.666663 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:30.666666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:30.666668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:30.666694 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:30.667915 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.668391 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.668397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.668399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.668474 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.668773 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.668777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.668779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.668933 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:30.669030 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.669452 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.669559 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:30.670312 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.659561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:44717 (local address 127.10.71.130:47967)
0619 14:57:30.659925 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.659957 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.660991 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.661469 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.661474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.661808 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:30.662321 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.662333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.663098 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.663103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.663929 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.663938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.664450 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.664454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.664593 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.665035 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:30.665065 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:30.666514 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:30.668030 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:30.668035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:30.668046 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:30.668277 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:30.668565 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:30.668567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:30.668569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:30.668688 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:30.669028 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:30.669033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:30.669240 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.669412 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.669660 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260619 14:57:30.671335 14960 heartbeater.cc:347] Connected to a master server at 127.10.71.190:44717
I20260619 14:57:30.671581 14960 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 14:57:30.672008 14960 heartbeater.cc:511] Master 127.10.71.190:44717 requested a full tablet report, sending...
I20260619 14:57:30.673015 14637 ts_manager.cc:194] Registered new tserver with Master: 277f0ed7eb5046179e424d286c85056e (127.10.71.130:44693)
I20260619 14:57:30.673655 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:47967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:30.684772 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:44717
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:30.782940 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:30.783178 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:30.783246 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:30.786881 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:30.786949 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:30.787024 14965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:30.791553 14965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:44717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:30.792435 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:30.793287 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:30.801355 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.801388 14965 server_base.cc:1034] running on GCE node
W20260619 14:57:30.801329 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:30.801342 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:30.801920 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:30.802469 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:30.803642 14965 hybrid_clock.cc:648] HybridClock initialized: now 1781881050803618 us; error 42 us; skew 500 ppm
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:30.806383 14965 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:30.806994 14965 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:30.807082 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:30.807255 14965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:30.809038 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "4874c0a882a14b41b571f153fed0d2ef"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "a3d2cee99239fcde33cca99341dc6753"
server_key_iv: "45c70f1e18684db432d228472c8efabf"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.809543 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "4874c0a882a14b41b571f153fed0d2ef"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "a3d2cee99239fcde33cca99341dc6753"
server_key_iv: "45c70f1e18684db432d228472c8efabf"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.814038 14965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:57:30.817065 14979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.818439 14965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:30.818583 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "4874c0a882a14b41b571f153fed0d2ef"
format_stamp: "Formatted at 2026-06-19 14:57:30 on dist-test-slave-3qdh"
server_key: "a3d2cee99239fcde33cca99341dc6753"
server_key_iv: "45c70f1e18684db432d228472c8efabf"
server_key_version: "encryptionkey@0"
I20260619 14:57:30.818706 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:30.828034 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:30.830559 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:30.830780 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:30.831305 14965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:30.832005 14965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:30.832075 14965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.832136 14965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:30.832185 14965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:30.841190 14965 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:41297
I20260619 14:57:30.841248 15092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:41297 every 8 connection(s)
I20260619 14:57:30.841742 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:30.848966 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 14965
I20260619 14:57:30.849090 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:30.849346 10526 external_mini_cluster.cc:1428] Setting key 89f8e4c3b813d6f419e683b96bf64d79
Jun 19 14:57:30 dist-test-slave-3qdh krb5kdc[14593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:30.853643 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.843204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:51729 (local address 127.10.71.190:44717)
0619 14:57:30.843348 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.843354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.844118 (+ 764us) server_negotiation.cc:407] Connection header received
0619 14:57:30.845210 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.845214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.845262 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.845338 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:30.846288 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.846795 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.847806 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.848023 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.848129 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.850585 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:30.850588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:30.850590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:30.850611 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:30.851882 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.852399 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.852402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.852403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.852467 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.852774 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.852776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.852777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.852947 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:30.853016 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.853438 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.853522 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:57:30.854295 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.843573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:44717 (local address 127.10.71.131:51729)
0619 14:57:30.843961 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.843997 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.845015 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.845494 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.845500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.845780 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:30.846157 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.846169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.846900 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.846902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.847682 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.847690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.848165 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.848168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.848292 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.848902 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:30.848922 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:30.850429 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:30.851979 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:30.851983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:30.851991 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:30.852248 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:30.852540 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:30.852544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:30.852546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:30.852688 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:30.853032 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:30.853036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:30.853238 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.853382 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.853595 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
I20260619 14:57:30.855239 15093 heartbeater.cc:347] Connected to a master server at 127.10.71.190:44717
I20260619 14:57:30.855400 15093 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:30.855697 15093 heartbeater.cc:511] Master 127.10.71.190:44717 requested a full tablet report, sending...
I20260619 14:57:30.856504 14637 ts_manager.cc:194] Registered new tserver with Master: 4874c0a882a14b41b571f153fed0d2ef (127.10.71.131:41297)
I20260619 14:57:30.856947 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:51729
I20260619 14:57:30.862864 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:57:30.870838 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.864629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.10.71.190:44717)
0619 14:57:30.864750 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.864755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.864952 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:57:30.865019 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.865021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.865072 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.865134 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:30.865688 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.866111 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.866781 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.866953 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.867021 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.868081 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:30.868084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:30.868085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:30.868104 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:30.869487 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.869853 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.869855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.869856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.869918 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:30.870151 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:30.870153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:30.870154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:30.870319 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:30.870412 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.870582 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.870658 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:30.873208 14637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:30.875801 14637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:30.888767 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.883011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:44693 (local address 127.0.0.1:37380)
0619 14:57:30.883703 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.883747 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.883934 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.884693 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.884695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.884708 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:30.884844 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.884853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.885905 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.885909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.887273 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.887283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.888234 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.888238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.888372 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.888414 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.888535 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.888640 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":61,"thread_start_us":78,"threads_started":1}
I20260619 14:57:30.888787 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.883149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.10.71.130:44693)
0619 14:57:30.883549 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.883560 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.883835 (+ 275us) server_negotiation.cc:407] Connection header received
0619 14:57:30.884124 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.884130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.884354 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.884480 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:30.884993 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.885773 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.887396 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.888115 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.888234 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.888382 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.888568 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.888654 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":118,"threads_started":1}
I20260619 14:57:30.889292 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.883326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.10.71.129:43359)
0619 14:57:30.883726 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.883739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.884135 (+ 396us) server_negotiation.cc:407] Connection header received
0619 14:57:30.884217 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.884224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.884440 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.884605 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:30.884909 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.886017 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.887555 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.888288 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.888411 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.888530 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.889050 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.889151 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":76,"threads_started":1}
I20260619 14:57:30.888967 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.883167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:43359 (local address 127.0.0.1:36068)
0619 14:57:30.883918 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.883930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.884020 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.884609 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.884614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.884639 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:30.884790 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.884798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.886196 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.886200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.887440 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.887447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.888594 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.888597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.888687 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.888708 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.888768 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.888838 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260619 14:57:30.889408 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.884268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.10.71.131:41297)
0619 14:57:30.884920 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.884930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.884949 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:57:30.885016 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.885020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.885163 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.885290 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:30.885492 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.886413 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.887831 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.888582 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.888673 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.888790 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.889142 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.889237 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":148,"threads_started":1}
I20260619 14:57:30.889884 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.883934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:41297 (local address 127.0.0.1:32990)
0619 14:57:30.884635 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.884649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.884722 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.885299 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.885302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.885318 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:30.885391 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.885395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.886529 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.886535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.887702 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.887712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.888841 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.888844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.888991 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.889017 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.889094 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.889151 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260619 14:57:30.891605 14761 tablet_service.cc:1467] Processing CreateTablet for tablet 5f34a6c976474f4bb10dc9a547cf80bb (DEFAULT_TABLE table=test-table [id=f45a857649924654b2ca7eafb8a9ff38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:30.891772 14894 tablet_service.cc:1467] Processing CreateTablet for tablet 5f34a6c976474f4bb10dc9a547cf80bb (DEFAULT_TABLE table=test-table [id=f45a857649924654b2ca7eafb8a9ff38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:30.892128 15027 tablet_service.cc:1467] Processing CreateTablet for tablet 5f34a6c976474f4bb10dc9a547cf80bb (DEFAULT_TABLE table=test-table [id=f45a857649924654b2ca7eafb8a9ff38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:30.892513 14761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f34a6c976474f4bb10dc9a547cf80bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:30.892669 14894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f34a6c976474f4bb10dc9a547cf80bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:30.892958 15027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f34a6c976474f4bb10dc9a547cf80bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:30.897698 15110 tablet_bootstrap.cc:492] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: Bootstrap starting.
I20260619 14:57:30.899377 15111 tablet_bootstrap.cc:492] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Bootstrap starting.
I20260619 14:57:30.900378 15110 tablet_bootstrap.cc:654] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:30.900586 15112 tablet_bootstrap.cc:492] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Bootstrap starting.
I20260619 14:57:30.901028 15111 tablet_bootstrap.cc:654] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:30.901062 15110 log.cc:824] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:30.902101 15111 log.cc:824] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:30.902534 15112 tablet_bootstrap.cc:654] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:30.902668 15110 tablet_bootstrap.cc:492] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: No bootstrap required, opened a new log
I20260619 14:57:30.902853 15110 ts_tablet_manager.cc:1397] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:30.903767 15112 log.cc:824] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:30.904083 15111 tablet_bootstrap.cc:492] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: No bootstrap required, opened a new log
I20260619 14:57:30.904335 15111 ts_tablet_manager.cc:1397] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:30.905526 15112 tablet_bootstrap.cc:492] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: No bootstrap required, opened a new log
I20260619 14:57:30.905668 15112 ts_tablet_manager.cc:1397] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:30.906466 15110 raft_consensus.cc:348] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.906688 15110 raft_consensus.cc:374] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:30.906729 15110 raft_consensus.cc:729] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78f9565e5a5465a8970a74e9a780ae5, State: Initialized, Role: FOLLOWER
I20260619 14:57:30.906699 15111 raft_consensus.cc:348] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.906857 15111 raft_consensus.cc:374] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:30.906906 15111 raft_consensus.cc:729] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277f0ed7eb5046179e424d286c85056e, State: Initialized, Role: FOLLOWER
I20260619 14:57:30.907148 15110 consensus_queue.cc:260] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.907230 15111 consensus_queue.cc:260] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.907717 14827 heartbeater.cc:503] Master 127.10.71.190:44717 was elected leader, sending a full tablet report...
I20260619 14:57:30.908022 15110 ts_tablet_manager.cc:1428] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:57:30.908083 15111 ts_tablet_manager.cc:1428] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:57:30.908355 14960 heartbeater.cc:503] Master 127.10.71.190:44717 was elected leader, sending a full tablet report...
W20260619 14:57:30.908586 14961 tablet.cc:2369] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:30.908895 15112 raft_consensus.cc:348] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.909144 15112 raft_consensus.cc:374] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:30.909222 15112 raft_consensus.cc:729] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4874c0a882a14b41b571f153fed0d2ef, State: Initialized, Role: FOLLOWER
I20260619 14:57:30.909507 15112 consensus_queue.cc:260] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.910010 15093 heartbeater.cc:503] Master 127.10.71.190:44717 was elected leader, sending a full tablet report...
I20260619 14:57:30.910156 15112 ts_tablet_manager.cc:1428] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:57:30.910825 15117 raft_consensus.cc:482] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:30.910972 15117 raft_consensus.cc:504] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.912002 15117 leader_election.cc:288] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78f9565e5a5465a8970a74e9a780ae5 (127.10.71.129:43359), 4874c0a882a14b41b571f153fed0d2ef (127.10.71.131:41297)
I20260619 14:57:30.915639 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.912584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56061 (local address 127.10.71.129:43359)
0619 14:57:30.912689 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.912694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.912710 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:57:30.912738 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.912739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.912766 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.912841 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:30.913213 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.913727 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.914735 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.915114 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.915227 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.915271 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.915464 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.915521 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 14:57:30.915648 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.912275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:39187 (local address 127.10.71.131:41297)
0619 14:57:30.912431 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.912434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.912544 (+ 110us) server_negotiation.cc:407] Connection header received
0619 14:57:30.912696 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.912699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.912738 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.912810 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:30.913112 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.913697 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.914741 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.915110 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.915228 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.915274 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.915474 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.915529 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 14:57:30.915669 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.912136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:43359 (local address 127.10.71.130:56061)
0619 14:57:30.912356 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.912369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.912440 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.912942 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.912947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.912960 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:30.913105 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.913111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.913835 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.913840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.914626 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.914634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.915221 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.915225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.915347 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.915367 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.915450 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.915517 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":84}
I20260619 14:57:30.915704 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.912201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:41297 (local address 127.10.71.130:39187)
0619 14:57:30.912455 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.912474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:30.912583 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:30.912839 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:30.912841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:30.912854 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:30.912998 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.913004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.913808 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.913813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.914631 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:30.914639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.915223 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.915226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:30.915357 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.915379 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:30.915462 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:57:30.915527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":38,"thread_start_us":96,"threads_started":1}
I20260619 14:57:30.916196 15047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34a6c976474f4bb10dc9a547cf80bb" candidate_uuid: "277f0ed7eb5046179e424d286c85056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4874c0a882a14b41b571f153fed0d2ef" is_pre_election: true
I20260619 14:57:30.916209 14781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34a6c976474f4bb10dc9a547cf80bb" candidate_uuid: "277f0ed7eb5046179e424d286c85056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78f9565e5a5465a8970a74e9a780ae5" is_pre_election: true
I20260619 14:57:30.916428 15047 raft_consensus.cc:2454] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277f0ed7eb5046179e424d286c85056e in term 0.
I20260619 14:57:30.916450 14781 raft_consensus.cc:2454] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277f0ed7eb5046179e424d286c85056e in term 0.
I20260619 14:57:30.916854 14847 leader_election.cc:302] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277f0ed7eb5046179e424d286c85056e, 4874c0a882a14b41b571f153fed0d2ef; no voters:
I20260619 14:57:30.917080 15117 raft_consensus.cc:2781] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:30.917148 15117 raft_consensus.cc:482] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:30.917186 15117 raft_consensus.cc:3037] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:30.917992 15117 raft_consensus.cc:504] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.918262 15117 leader_election.cc:288] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [CANDIDATE]: Term 1 election: Requested vote from peers a78f9565e5a5465a8970a74e9a780ae5 (127.10.71.129:43359), 4874c0a882a14b41b571f153fed0d2ef (127.10.71.131:41297)
I20260619 14:57:30.918527 14781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34a6c976474f4bb10dc9a547cf80bb" candidate_uuid: "277f0ed7eb5046179e424d286c85056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78f9565e5a5465a8970a74e9a780ae5"
I20260619 14:57:30.918527 15047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34a6c976474f4bb10dc9a547cf80bb" candidate_uuid: "277f0ed7eb5046179e424d286c85056e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4874c0a882a14b41b571f153fed0d2ef"
I20260619 14:57:30.918633 15047 raft_consensus.cc:3037] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:30.918633 14781 raft_consensus.cc:3037] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:30.919711 14781 raft_consensus.cc:2454] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277f0ed7eb5046179e424d286c85056e in term 1.
I20260619 14:57:30.919711 15047 raft_consensus.cc:2454] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277f0ed7eb5046179e424d286c85056e in term 1.
I20260619 14:57:30.919948 14847 leader_election.cc:302] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277f0ed7eb5046179e424d286c85056e, 4874c0a882a14b41b571f153fed0d2ef; no voters:
I20260619 14:57:30.920145 15117 raft_consensus.cc:2781] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:30.920276 15117 raft_consensus.cc:686] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 1 LEADER]: Becoming Leader. State: Replica: 277f0ed7eb5046179e424d286c85056e, State: Running, Role: LEADER
I20260619 14:57:30.920586 15117 consensus_queue.cc:237] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } }
I20260619 14:57:30.923075 14637 catalog_manager.cc:5495] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e reported cstate change: term changed from 0 to 1, leader changed from <none> to 277f0ed7eb5046179e424d286c85056e (127.10.71.130). New cstate: current_term: 1 leader_uuid: "277f0ed7eb5046179e424d286c85056e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277f0ed7eb5046179e424d286c85056e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 44693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:30.936089 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.933190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.10.71.130:44693)
0619 14:57:30.933284 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:30.933287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:30.933363 (+ 76us) server_negotiation.cc:407] Connection header received
0619 14:57:30.933473 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:30.933474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:30.933514 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:30.933592 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:30.933806 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.934234 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.935030 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:30.935221 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:30.935283 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:30.935510 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:30.935802 (+ 292us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:30.935871 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:30.935926 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:57:30.935960 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:57:30.940413 15047 raft_consensus.cc:1261] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 1 FOLLOWER]: Refusing update from remote peer 277f0ed7eb5046179e424d286c85056e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:30.940433 14781 raft_consensus.cc:1261] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5 [term 1 FOLLOWER]: Refusing update from remote peer 277f0ed7eb5046179e424d286c85056e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:30.940989 15120 consensus_queue.cc:1035] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78f9565e5a5465a8970a74e9a780ae5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:30.941149 15117 consensus_queue.cc:1035] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4874c0a882a14b41b571f153fed0d2ef" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:30.954334 14636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:30.954476 14636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:30.956398 14636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299ecf53d87b447f8a1031619d51b880: Sending DeleteTablet for 3 replicas of tablet 5f34a6c976474f4bb10dc9a547cf80bb
I20260619 14:57:30.957008 14894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f34a6c976474f4bb10dc9a547cf80bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:30 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:37380
I20260619 14:57:30.957052 14761 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f34a6c976474f4bb10dc9a547cf80bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:30 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:36068
I20260619 14:57:30.957185 15027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f34a6c976474f4bb10dc9a547cf80bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:30 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:32990
I20260619 14:57:30.957480 15132 tablet_replica.cc:330] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: stopping tablet replica
I20260619 14:57:30.957532 15134 tablet_replica.cc:330] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: stopping tablet replica
I20260619 14:57:30.957757 15132 raft_consensus.cc:2229] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:30.957839 15134 raft_consensus.cc:2229] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:30.958098 15132 raft_consensus.cc:2258] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:30.958127 15133 tablet_replica.cc:330] T 5f34a6c976474f4bb10dc9a547cf80bb P a78f9565e5a5465a8970a74e9a780ae5: stopping tablet replica
I20260619 14:57:30.958217 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14699
I20260619 14:57:30.960626 15132 ts_tablet_manager.cc:1905] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:30.962909 15132 ts_tablet_manager.cc:1918] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:30.962998 15132 log.cc:1196] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/5f34a6c976474f4bb10dc9a547cf80bb
I20260619 14:57:30.963303 15132 ts_tablet_manager.cc:1939] T 5f34a6c976474f4bb10dc9a547cf80bb P 277f0ed7eb5046179e424d286c85056e: Deleting consensus metadata
I20260619 14:57:30.964277 14624 catalog_manager.cc:4841] TS 277f0ed7eb5046179e424d286c85056e (127.10.71.130:44693): tablet 5f34a6c976474f4bb10dc9a547cf80bb (table test-table [id=f45a857649924654b2ca7eafb8a9ff38]) successfully deleted
W20260619 14:57:30.964759 14622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:43359 (error 108)
I20260619 14:57:30.965031 15134 raft_consensus.cc:2258] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:30.965631 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:30.965392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:43359 (local address 127.0.0.1:36072)
0619 14:57:30.965499 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:30.965559 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:43359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:57:30.965763 14622 catalog_manager.cc:4584] TS a78f9565e5a5465a8970a74e9a780ae5 (127.10.71.129:43359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f34a6c976474f4bb10dc9a547cf80bb: Network error: Client connection negotiation failed: client connection to 127.10.71.129:43359: connect: Connection refused (error 111)
I20260619 14:57:30.965854 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14832
I20260619 14:57:30.967295 15134 ts_tablet_manager.cc:1905] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:30.969353 15134 ts_tablet_manager.cc:1918] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:30.969466 15134 log.cc:1196] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/5f34a6c976474f4bb10dc9a547cf80bb
I20260619 14:57:30.969803 15134 ts_tablet_manager.cc:1939] T 5f34a6c976474f4bb10dc9a547cf80bb P 4874c0a882a14b41b571f153fed0d2ef: Deleting consensus metadata
I20260619 14:57:30.970675 14621 catalog_manager.cc:4841] TS 4874c0a882a14b41b571f153fed0d2ef (127.10.71.131:41297): tablet 5f34a6c976474f4bb10dc9a547cf80bb (table test-table [id=f45a857649924654b2ca7eafb8a9ff38]) successfully deleted
I20260619 14:57:30.971644 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14965
I20260619 14:57:30.976562 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 14606
2026-06-19T14:57:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1967 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:31Z Disabled control of system clock
I20260619 14:57:31.009433 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34895
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:57:31.119122 15139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:31.119323 15139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:31.119356 15139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:31.123068 15139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:31.123111 15139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:31.123127 15139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:31.123143 15139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:31.123158 15139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:31.127532 15139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:34895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:31.128356 15139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:31.129477 15139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:31.136891 15147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.137001 15144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.137099 15139 server_base.cc:1034] running on GCE node
W20260619 14:57:31.137007 15145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.138190 15139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:31.139032 15139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:31.140189 15139 hybrid_clock.cc:648] HybridClock initialized: now 1781881051140162 us; error 39 us; skew 500 ppm
I20260619 14:57:31.141736 15139 webserver.cc:460] Webserver started at http://127.10.71.190:42981/ using document root <none> and password file <none>
I20260619 14:57:31.142239 15139 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:31.142297 15139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:31.142493 15139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:31.144244 15139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "67aebffd587e4b56934978a027db43cf"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "375b72e9f7e376011e04c583614a7e18"
server_key_iv: "fa63658a81a12bc700e8551e62e2d04e"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.144804 15139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "67aebffd587e4b56934978a027db43cf"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "375b72e9f7e376011e04c583614a7e18"
server_key_iv: "fa63658a81a12bc700e8551e62e2d04e"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.147838 15139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:31.149979 15153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.151026 15139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:31.151171 15139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "67aebffd587e4b56934978a027db43cf"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "375b72e9f7e376011e04c583614a7e18"
server_key_iv: "fa63658a81a12bc700e8551e62e2d04e"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.151266 15139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:31.155288 15139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:31.155798 15139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:31.155956 15139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:31.162062 15205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:34895 every 8 connection(s)
I20260619 14:57:31.162063 15139 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:34895
I20260619 14:57:31.162833 15139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:31.164347 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15139
I20260619 14:57:31.164481 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:31.164721 10526 external_mini_cluster.cc:1428] Setting key 1d7158c3ddc95c2b342eefa94b605432
I20260619 14:57:31.165748 15206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:31.169636 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.165989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.10.71.190:34895)
0619 14:57:31.166543 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.166556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.166594 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:57:31.167130 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.167143 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.167420 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.167715 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.168536 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.168541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.168566 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.168576 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.168623 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.168638 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.168779 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.168846 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.169063 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260619 14:57:31.171013 15206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf: Bootstrap starting.
I20260619 14:57:31.173041 15206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:31.173637 15206 log.cc:824] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:31.175305 15206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf: No bootstrap required, opened a new log
I20260619 14:57:31.178763 15206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aebffd587e4b56934978a027db43cf" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34895 } }
I20260619 14:57:31.178929 15206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:31.178960 15206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67aebffd587e4b56934978a027db43cf, State: Initialized, Role: FOLLOWER
I20260619 14:57:31.179253 15206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aebffd587e4b56934978a027db43cf" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34895 } }
I20260619 14:57:31.179345 15206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:31.179385 15206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:31.179445 15206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:31.180429 15206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aebffd587e4b56934978a027db43cf" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34895 } }
I20260619 14:57:31.180683 15206 leader_election.cc:302] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67aebffd587e4b56934978a027db43cf; no voters:
I20260619 14:57:31.180917 15206 leader_election.cc:288] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:31.181051 15211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:31.181290 15211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [term 1 LEADER]: Becoming Leader. State: Replica: 67aebffd587e4b56934978a027db43cf, State: Running, Role: LEADER
I20260619 14:57:31.181545 15211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aebffd587e4b56934978a027db43cf" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34895 } }
I20260619 14:57:31.181939 15206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:31.182960 15213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67aebffd587e4b56934978a027db43cf. Latest consensus state: current_term: 1 leader_uuid: "67aebffd587e4b56934978a027db43cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aebffd587e4b56934978a027db43cf" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34895 } } }
I20260619 14:57:31.183084 15213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:31.183344 15212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67aebffd587e4b56934978a027db43cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aebffd587e4b56934978a027db43cf" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34895 } } }
I20260619 14:57:31.183426 15212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:31.183519 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:31.186296 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:31.190982 15220 catalog_manager.cc:1347] Generated new cluster ID: 8a7bd2f68d1444b4a721be4f3968de10
I20260619 14:57:31.191036 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:31.206316 15220 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:31.206570 15220 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:31.212414 15220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf: Generated new TSK 0
I20260619 14:57:31.212935 15220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:57:31.215291 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34895
--builtin_ntp_servers=127.10.71.148:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:57:31.329731 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:31.329928 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:31.329962 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:31.333808 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:31.333851 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:31.333937 15230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:31.338187 15230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:31.339016 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:31.340174 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:31.348397 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.348414 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.348397 15230 server_base.cc:1034] running on GCE node
W20260619 14:57:31.348389 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.348941 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:31.349476 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:31.350651 15230 hybrid_clock.cc:648] HybridClock initialized: now 1781881051350643 us; error 27 us; skew 500 ppm
I20260619 14:57:31.352088 15230 webserver.cc:460] Webserver started at http://127.10.71.129:37449/ using document root <none> and password file <none>
I20260619 14:57:31.352651 15230 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:31.352735 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:31.352945 15230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:31.354789 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "c807036788de4752b66c9e394eacf899"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "5160a95acea75950cc1bf7ddefeb31f9"
server_key_iv: "e901174c0738863b5c0afcb76d384b66"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.355278 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "c807036788de4752b66c9e394eacf899"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "5160a95acea75950cc1bf7ddefeb31f9"
server_key_iv: "e901174c0738863b5c0afcb76d384b66"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.358361 15230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:31.360476 15244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.361485 15230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:31.361603 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "c807036788de4752b66c9e394eacf899"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "5160a95acea75950cc1bf7ddefeb31f9"
server_key_iv: "e901174c0738863b5c0afcb76d384b66"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.361702 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:31.365806 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:31.366303 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:31.366460 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:31.366897 15230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:31.367626 15230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:31.367690 15230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.367745 15230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:31.367806 15230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.376070 15230 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:37659
I20260619 14:57:31.376091 15357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:37659 every 8 connection(s)
I20260619 14:57:31.376854 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:31.380800 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15230
I20260619 14:57:31.380911 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:31.381206 10526 external_mini_cluster.cc:1428] Setting key 7b4a8370e48d737ae631ddf7c5c11bd3
I20260619 14:57:31.382370 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.378316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:53239 (local address 127.10.71.190:34895)
0619 14:57:31.378437 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.378440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.379289 (+ 849us) server_negotiation.cc:407] Connection header received
0619 14:57:31.380501 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.380504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.380550 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.380639 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.381832 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.381835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.381838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.381850 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.381871 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.381881 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.381972 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.382164 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.382209 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:31.383092 15360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.378557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34895 (local address 127.10.71.129:53239)
0619 14:57:31.379137 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.379183 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.380253 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.380824 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.380831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.380960 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.381530 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.381531 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.381588 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.381668 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.381982 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.381990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.381995 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.382152 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.382350 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
I20260619 14:57:31.383347 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34895
--builtin_ntp_servers=127.10.71.148:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:57:31.384095 15358 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34895
I20260619 14:57:31.384274 15358 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:31.384646 15358 heartbeater.cc:511] Master 127.10.71.190:34895 requested a full tablet report, sending...
I20260619 14:57:31.386020 15170 ts_manager.cc:194] Registered new tserver with Master: c807036788de4752b66c9e394eacf899 (127.10.71.129:37659)
W20260619 14:57:31.481817 15361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:31.482002 15361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:31.482038 15361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:31.486002 15361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:31.486043 15361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:31.486097 15361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:31.490310 15361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:31.491046 15361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:31.492066 15361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:31.500384 15367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.500384 15366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.500404 15369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.500644 15361 server_base.cc:1034] running on GCE node
I20260619 14:57:31.501024 15361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:31.501544 15361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:31.502728 15361 hybrid_clock.cc:648] HybridClock initialized: now 1781881051502730 us; error 33 us; skew 500 ppm
I20260619 14:57:31.504406 15361 webserver.cc:460] Webserver started at http://127.10.71.130:46261/ using document root <none> and password file <none>
I20260619 14:57:31.504909 15361 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:31.504994 15361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:31.505187 15361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:31.506857 15361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "b4393e62d29147dcb114f70d8f8b5411"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "4f39e31b1b5d0c9896443910cfe95346"
server_key_iv: "c5b54f6649b81cf8ce0dcd191b172cd8"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.507339 15361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "b4393e62d29147dcb114f70d8f8b5411"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "4f39e31b1b5d0c9896443910cfe95346"
server_key_iv: "c5b54f6649b81cf8ce0dcd191b172cd8"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.510455 15361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 14:57:31.512454 15375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.513445 15361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:31.513568 15361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "b4393e62d29147dcb114f70d8f8b5411"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "4f39e31b1b5d0c9896443910cfe95346"
server_key_iv: "c5b54f6649b81cf8ce0dcd191b172cd8"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.513670 15361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:31.517427 15361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:31.517874 15361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:31.518020 15361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:31.518464 15361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:31.519183 15361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:31.519259 15361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.519326 15361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:31.519356 15361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.526664 15361 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:32797
I20260619 14:57:31.526692 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:32797 every 8 connection(s)
I20260619 14:57:31.527333 15361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:31.529961 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15361
I20260619 14:57:31.530079 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:31.530311 10526 external_mini_cluster.cc:1428] Setting key 6513c931317726b2bc6e133ae5c3796c
I20260619 14:57:31.532212 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34895
--builtin_ntp_servers=127.10.71.148:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:57:31.532500 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.528769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:48503 (local address 127.10.71.190:34895)
0619 14:57:31.528912 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.528916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.529717 (+ 801us) server_negotiation.cc:407] Connection header received
0619 14:57:31.530949 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.530952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.530993 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.531055 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.532075 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.532081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.532085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.532099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.532123 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.532133 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.532218 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.532345 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.532384 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:57:31.533236 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.529065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34895 (local address 127.10.71.130:48503)
0619 14:57:31.529594 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.529625 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.530781 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.531222 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.531229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.531318 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.531833 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.531835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.531864 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.531933 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.532211 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.532217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.532220 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.532381 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.532625 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260619 14:57:31.534255 15489 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34895
I20260619 14:57:31.534447 15489 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:31.534759 15489 heartbeater.cc:511] Master 127.10.71.190:34895 requested a full tablet report, sending...
I20260619 14:57:31.535625 15170 ts_manager.cc:194] Registered new tserver with Master: b4393e62d29147dcb114f70d8f8b5411 (127.10.71.130:32797)
W20260619 14:57:31.632434 15492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:31.632648 15492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:31.632716 15492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:31.636255 15492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:31.636348 15492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:31.636441 15492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:31.640552 15492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:31.641311 15492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:31.642405 15492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:31.651012 15500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.651007 15497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.651007 15498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.651203 15492 server_base.cc:1034] running on GCE node
I20260619 14:57:31.651614 15492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:31.652076 15492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:31.653227 15492 hybrid_clock.cc:648] HybridClock initialized: now 1781881051653211 us; error 30 us; skew 500 ppm
I20260619 14:57:31.655010 15492 webserver.cc:460] Webserver started at http://127.10.71.131:33495/ using document root <none> and password file <none>
I20260619 14:57:31.655514 15492 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:31.655602 15492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:31.655740 15492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:31.657389 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "9344f87b26e44a71ac82d06d1188b325"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "a7fe7bb5d964ce3301cd376ce54b95cf"
server_key_iv: "4016b821dfc51f22fc249c1e7eff3e48"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.657887 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "9344f87b26e44a71ac82d06d1188b325"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "a7fe7bb5d964ce3301cd376ce54b95cf"
server_key_iv: "4016b821dfc51f22fc249c1e7eff3e48"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.660998 15492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:57:31.662907 15506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.663990 15492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:31.664130 15492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "9344f87b26e44a71ac82d06d1188b325"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "a7fe7bb5d964ce3301cd376ce54b95cf"
server_key_iv: "4016b821dfc51f22fc249c1e7eff3e48"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.664270 15492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:31.668035 15492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:31.668596 15492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:31.668747 15492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:31.669281 15492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:31.669970 15492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:31.670028 15492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.670079 15492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:31.670123 15492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.677441 15492 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:38777
I20260619 14:57:31.677495 15619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:38777 every 8 connection(s)
I20260619 14:57:31.678130 15492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:31.682832 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.679519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:37113 (local address 127.10.71.190:34895)
0619 14:57:31.679654 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.679659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.680578 (+ 919us) server_negotiation.cc:407] Connection header received
0619 14:57:31.681528 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.681530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.681562 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.681629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.682467 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.682469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.682471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.682481 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.682500 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.682507 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.682579 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.682708 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.682739 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:57:31.683633 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.679869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34895 (local address 127.10.71.131:37113)
0619 14:57:31.680466 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.680496 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.681384 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.681724 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.681729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.681795 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.682261 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.682263 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.682292 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.682342 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.682586 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.682592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.682597 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.682739 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.682937 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
I20260619 14:57:31.684762 15620 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34895
I20260619 14:57:31.684962 15620 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:31.685305 15620 heartbeater.cc:511] Master 127.10.71.190:34895 requested a full tablet report, sending...
I20260619 14:57:31.686257 15170 ts_manager.cc:194] Registered new tserver with Master: 9344f87b26e44a71ac82d06d1188b325 (127.10.71.131:38777)
I20260619 14:57:31.687505 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15492
I20260619 14:57:31.687623 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:31.687855 10526 external_mini_cluster.cc:1428] Setting key 8dd4519ff34ee4192be71d46cf61bfe5
I20260619 14:57:31.689688 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:57:31.692502 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.691313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.10.71.190:34895)
0619 14:57:31.691429 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.691432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.691567 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:57:31.691656 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.691658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.691680 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.691735 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.692230 (+ 495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.692233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.692234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.692241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.692254 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.692260 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.692364 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.692416 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.692439 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:57:31.694764 15170 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:31.697175 15170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:31.706604 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.704028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:38777 (local address 127.0.0.1:57056)
0619 14:57:31.704671 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.704682 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.704756 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.705209 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.705212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.705224 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.705626 (+ 402us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.705627 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.705637 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.705659 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.706317 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.706325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.706329 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.706416 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.706471 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":78,"threads_started":1}
I20260619 14:57:31.706689 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.704005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.10.71.129:37659)
0619 14:57:31.704494 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.704506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.704530 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:57:31.704585 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.704593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.704764 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.705260 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.706198 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.706205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.706215 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.706226 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.706261 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.706284 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.706421 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.706472 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.706539 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":166,"threads_started":1}
I20260619 14:57:31.706887 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.704496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.10.71.131:38777)
0619 14:57:31.704878 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.704888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.704904 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:57:31.704942 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.704946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.705083 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.705404 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.706149 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.706168 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.706178 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.706189 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.706226 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.706241 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.706336 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.706679 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.706752 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":139,"threads_started":1}
I20260619 14:57:31.706604 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.703782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37659 (local address 127.0.0.1:39818)
0619 14:57:31.704382 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.704430 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.704508 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.704940 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.704946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.705008 (+ 62us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.705466 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.705470 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.705489 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.705547 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.706395 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.706400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.706403 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.706453 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.706472 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":73,"threads_started":1}
I20260619 14:57:31.707579 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.704432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:32797 (local address 127.0.0.1:60590)
0619 14:57:31.705020 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.705032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.705099 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.706026 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.706029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.706042 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.706913 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.706915 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.706943 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.706971 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.707347 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.707349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.707351 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.707439 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.707472 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":49,"threads_started":1}
I20260619 14:57:31.707810 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.704797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.10.71.130:32797)
0619 14:57:31.705635 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.705651 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.705672 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:57:31.705720 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.705725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.705879 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.706011 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.707118 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.707123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.707132 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.707147 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.707192 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.707217 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.707303 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.707455 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.707511 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260619 14:57:31.709352 15554 tablet_service.cc:1467] Processing CreateTablet for tablet afd7ae00c9014faabd2d196a9524ccdb (DEFAULT_TABLE table=test-table [id=a11be46859434e89a7dc6775fba4ca98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:31.709354 15292 tablet_service.cc:1467] Processing CreateTablet for tablet afd7ae00c9014faabd2d196a9524ccdb (DEFAULT_TABLE table=test-table [id=a11be46859434e89a7dc6775fba4ca98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:31.709949 15423 tablet_service.cc:1467] Processing CreateTablet for tablet afd7ae00c9014faabd2d196a9524ccdb (DEFAULT_TABLE table=test-table [id=a11be46859434e89a7dc6775fba4ca98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:31.710300 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd7ae00c9014faabd2d196a9524ccdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:31.710300 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd7ae00c9014faabd2d196a9524ccdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:31.710759 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd7ae00c9014faabd2d196a9524ccdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:31.716276 15634 tablet_bootstrap.cc:492] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Bootstrap starting.
I20260619 14:57:31.716995 15636 tablet_bootstrap.cc:492] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Bootstrap starting.
I20260619 14:57:31.717394 15635 tablet_bootstrap.cc:492] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899: Bootstrap starting.
I20260619 14:57:31.718456 15634 tablet_bootstrap.cc:654] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:31.718914 15636 tablet_bootstrap.cc:654] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:31.719174 15635 tablet_bootstrap.cc:654] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:31.719281 15634 log.cc:824] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:31.719652 15636 log.cc:824] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:31.719839 15635 log.cc:824] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:31.721105 15634 tablet_bootstrap.cc:492] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: No bootstrap required, opened a new log
I20260619 14:57:31.721181 15636 tablet_bootstrap.cc:492] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: No bootstrap required, opened a new log
I20260619 14:57:31.721215 15635 tablet_bootstrap.cc:492] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899: No bootstrap required, opened a new log
I20260619 14:57:31.721323 15636 ts_tablet_manager.cc:1397] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:57:31.721315 15634 ts_tablet_manager.cc:1397] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 14:57:31.721378 15635 ts_tablet_manager.cc:1397] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:57:31.724124 15635 raft_consensus.cc:348] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.724364 15635 raft_consensus.cc:374] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:31.724411 15635 raft_consensus.cc:729] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c807036788de4752b66c9e394eacf899, State: Initialized, Role: FOLLOWER
I20260619 14:57:31.724818 15635 consensus_queue.cc:260] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.725418 15634 raft_consensus.cc:348] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.725418 15636 raft_consensus.cc:348] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.725642 15634 raft_consensus.cc:374] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:31.725642 15636 raft_consensus.cc:374] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:31.725690 15634 raft_consensus.cc:729] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4393e62d29147dcb114f70d8f8b5411, State: Initialized, Role: FOLLOWER
I20260619 14:57:31.725697 15636 raft_consensus.cc:729] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9344f87b26e44a71ac82d06d1188b325, State: Initialized, Role: FOLLOWER
I20260619 14:57:31.725739 15635 ts_tablet_manager.cc:1428] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260619 14:57:31.726060 15358 heartbeater.cc:503] Master 127.10.71.190:34895 was elected leader, sending a full tablet report...
I20260619 14:57:31.726097 15636 consensus_queue.cc:260] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.726097 15634 consensus_queue.cc:260] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.726828 15620 heartbeater.cc:503] Master 127.10.71.190:34895 was elected leader, sending a full tablet report...
I20260619 14:57:31.726945 15636 ts_tablet_manager.cc:1428] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:57:31.726945 15634 ts_tablet_manager.cc:1428] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:57:31.727248 15489 heartbeater.cc:503] Master 127.10.71.190:34895 was elected leader, sending a full tablet report...
I20260619 14:57:31.737236 15641 raft_consensus.cc:482] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:31.737342 15641 raft_consensus.cc:504] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.738109 15641 leader_election.cc:288] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c807036788de4752b66c9e394eacf899 (127.10.71.129:37659), 9344f87b26e44a71ac82d06d1188b325 (127.10.71.131:38777)
I20260619 14:57:31.739759 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.738364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:52329 (local address 127.10.71.131:38777)
0619 14:57:31.738444 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.738448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.738747 (+ 299us) server_negotiation.cc:407] Connection header received
0619 14:57:31.738820 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.738821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.738841 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.738914 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.739445 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.739447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.739449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.739459 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.739475 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.739482 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.739546 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.739636 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.739667 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 14:57:31.739791 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.738296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:38777 (local address 127.10.71.130:52329)
0619 14:57:31.738658 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.738672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.738745 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.738932 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.738934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.738949 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.739318 (+ 369us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.739320 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.739329 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.739350 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.739549 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.739550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.739552 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.739630 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.739662 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":88,"threads_started":1}
I20260619 14:57:31.739814 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.738267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:37243 (local address 127.10.71.129:37659)
0619 14:57:31.738358 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.738361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.738467 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:57:31.738578 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.738580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.738603 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.738658 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.739311 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.739315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.739317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.739328 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.739344 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.739352 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.739431 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.739682 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.739718 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:57:31.739811 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.738232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37659 (local address 127.10.71.130:37243)
0619 14:57:31.738400 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.738417 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:31.738493 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:31.738704 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:31.738709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:31.738726 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:31.739158 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:57:31.739161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:31.739174 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:57:31.739206 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:31.739582 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:31.739584 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:31.739586 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 14:57:31.739677 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:57:31.739711 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 14:57:31.740361 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd7ae00c9014faabd2d196a9524ccdb" candidate_uuid: "b4393e62d29147dcb114f70d8f8b5411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c807036788de4752b66c9e394eacf899" is_pre_election: true
I20260619 14:57:31.740403 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd7ae00c9014faabd2d196a9524ccdb" candidate_uuid: "b4393e62d29147dcb114f70d8f8b5411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9344f87b26e44a71ac82d06d1188b325" is_pre_election: true
I20260619 14:57:31.740574 15312 raft_consensus.cc:2454] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4393e62d29147dcb114f70d8f8b5411 in term 0.
I20260619 14:57:31.740607 15574 raft_consensus.cc:2454] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4393e62d29147dcb114f70d8f8b5411 in term 0.
I20260619 14:57:31.741007 15377 leader_election.cc:302] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9344f87b26e44a71ac82d06d1188b325, b4393e62d29147dcb114f70d8f8b5411, c807036788de4752b66c9e394eacf899; no voters:
I20260619 14:57:31.741349 15641 raft_consensus.cc:2781] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:31.741408 15641 raft_consensus.cc:482] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:31.741480 15641 raft_consensus.cc:3037] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:31.742307 15641 raft_consensus.cc:504] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.742625 15641 leader_election.cc:288] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [CANDIDATE]: Term 1 election: Requested vote from peers c807036788de4752b66c9e394eacf899 (127.10.71.129:37659), 9344f87b26e44a71ac82d06d1188b325 (127.10.71.131:38777)
I20260619 14:57:31.742935 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd7ae00c9014faabd2d196a9524ccdb" candidate_uuid: "b4393e62d29147dcb114f70d8f8b5411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c807036788de4752b66c9e394eacf899"
I20260619 14:57:31.742952 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd7ae00c9014faabd2d196a9524ccdb" candidate_uuid: "b4393e62d29147dcb114f70d8f8b5411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9344f87b26e44a71ac82d06d1188b325"
I20260619 14:57:31.743103 15312 raft_consensus.cc:3037] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:31.743077 15574 raft_consensus.cc:3037] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:31.744316 15312 raft_consensus.cc:2454] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4393e62d29147dcb114f70d8f8b5411 in term 1.
I20260619 14:57:31.744333 15574 raft_consensus.cc:2454] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4393e62d29147dcb114f70d8f8b5411 in term 1.
I20260619 14:57:31.744628 15378 leader_election.cc:302] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9344f87b26e44a71ac82d06d1188b325, b4393e62d29147dcb114f70d8f8b5411; no voters:
I20260619 14:57:31.744796 15641 raft_consensus.cc:2781] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:31.745003 15641 raft_consensus.cc:686] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 1 LEADER]: Becoming Leader. State: Replica: b4393e62d29147dcb114f70d8f8b5411, State: Running, Role: LEADER
I20260619 14:57:31.745280 15641 consensus_queue.cc:237] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } }
I20260619 14:57:31.747710 15170 catalog_manager.cc:5495] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4393e62d29147dcb114f70d8f8b5411 (127.10.71.130). New cstate: current_term: 1 leader_uuid: "b4393e62d29147dcb114f70d8f8b5411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4393e62d29147dcb114f70d8f8b5411" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 32797 } health_report { overall_health: HEALTHY } } }
I20260619 14:57:31.756211 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.754873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.10.71.130:32797)
0619 14:57:31.754984 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:31.754988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:31.755036 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:57:31.755183 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:31.755186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:31.755237 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:31.755309 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:31.755875 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:31.755878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:31.755880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:31.755890 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:31.755904 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:31.755912 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:31.756005 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:31.756081 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 14:57:31.756111 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:57:31.760671 15312 raft_consensus.cc:1261] T afd7ae00c9014faabd2d196a9524ccdb P c807036788de4752b66c9e394eacf899 [term 1 FOLLOWER]: Refusing update from remote peer b4393e62d29147dcb114f70d8f8b5411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:31.760668 15574 raft_consensus.cc:1261] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 1 FOLLOWER]: Refusing update from remote peer b4393e62d29147dcb114f70d8f8b5411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:31.761281 15644 consensus_queue.cc:1035] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c807036788de4752b66c9e394eacf899" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:31.761477 15641 consensus_queue.cc:1035] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9344f87b26e44a71ac82d06d1188b325" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:31.767956 15649 mvcc.cc:205] Tried to move back new op lower bound from 7298584788005969920 to 7298584787950878720. Current Snapshot: MvccSnapshot[applied={T|T < 7298584788005969920}]
I20260619 14:57:31.769670 15648 mvcc.cc:205] Tried to move back new op lower bound from 7298584788005969920 to 7298584787950878720. Current Snapshot: MvccSnapshot[applied={T|T < 7298584788005969920}]
I20260619 14:57:31.775267 15168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:31.775395 15168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:31.777456 15168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67aebffd587e4b56934978a027db43cf: Sending DeleteTablet for 3 replicas of tablet afd7ae00c9014faabd2d196a9524ccdb
I20260619 14:57:31.778054 15292 tablet_service.cc:1514] Processing DeleteTablet for tablet afd7ae00c9014faabd2d196a9524ccdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:31 UTC) from {username='slave'} at 127.0.0.1:39818
I20260619 14:57:31.778193 15554 tablet_service.cc:1514] Processing DeleteTablet for tablet afd7ae00c9014faabd2d196a9524ccdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:31 UTC) from {username='slave'} at 127.0.0.1:57056
I20260619 14:57:31.778573 15658 tablet_replica.cc:330] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: stopping tablet replica
I20260619 14:57:31.778811 15658 raft_consensus.cc:2229] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:31.778781 15423 tablet_service.cc:1514] Processing DeleteTablet for tablet afd7ae00c9014faabd2d196a9524ccdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:31 UTC) from {username='slave'} at 127.0.0.1:60590
I20260619 14:57:31.779032 15658 raft_consensus.cc:2258] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:31.779181 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15230
I20260619 14:57:31.779306 15659 tablet_replica.cc:330] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: stopping tablet replica
I20260619 14:57:31.779604 15659 raft_consensus.cc:2229] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:31.780314 15659 raft_consensus.cc:2258] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:31.781695 15659 ts_tablet_manager.cc:1905] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:31.784217 15659 ts_tablet_manager.cc:1918] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:31.784327 15659 log.cc:1196] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/afd7ae00c9014faabd2d196a9524ccdb
I20260619 14:57:31.784629 15659 ts_tablet_manager.cc:1939] T afd7ae00c9014faabd2d196a9524ccdb P b4393e62d29147dcb114f70d8f8b5411: Deleting consensus metadata
I20260619 14:57:31.785756 15156 catalog_manager.cc:4841] TS b4393e62d29147dcb114f70d8f8b5411 (127.10.71.130:32797): tablet afd7ae00c9014faabd2d196a9524ccdb (table test-table [id=a11be46859434e89a7dc6775fba4ca98]) successfully deleted
I20260619 14:57:31.786160 15658 ts_tablet_manager.cc:1905] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:57:31.786387 15155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:37659 (error 108)
I20260619 14:57:31.786585 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15361
I20260619 14:57:31.788658 15658 ts_tablet_manager.cc:1918] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:31.788749 15658 log.cc:1196] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/afd7ae00c9014faabd2d196a9524ccdb
I20260619 14:57:31.788875 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:31.788275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37659 (local address 127.0.0.1:39826)
0619 14:57:31.788695 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:31.788789 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260619 14:57:31.789028 15658 ts_tablet_manager.cc:1939] T afd7ae00c9014faabd2d196a9524ccdb P 9344f87b26e44a71ac82d06d1188b325: Deleting consensus metadata
W20260619 14:57:31.789323 15155 catalog_manager.cc:4584] TS c807036788de4752b66c9e394eacf899 (127.10.71.129:37659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afd7ae00c9014faabd2d196a9524ccdb: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37659: connect: Connection refused (error 111)
I20260619 14:57:31.790043 15156 catalog_manager.cc:4841] TS 9344f87b26e44a71ac82d06d1188b325 (127.10.71.131:38777): tablet afd7ae00c9014faabd2d196a9524ccdb (table test-table [id=a11be46859434e89a7dc6775fba4ca98]) successfully deleted
I20260619 14:57:31.792631 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15492
I20260619 14:57:31.797756 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15139
2026-06-19T14:57:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (819 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:57:31.820428 10526 mini_oidc.cc:152] Starting JWKS server
I20260619 14:57:31.821898 10526 webserver.cc:460] Webserver started at https://127.0.0.1:41855/ using document root <none> and password file <none>
I20260619 14:57:31.822084 10526 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:57:31.822363 10526 webserver.cc:460] Webserver started at http://127.0.0.1:34973/ using document root <none> and password file <none>
2026-06-19T14:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:31Z Disabled control of system clock
I20260619 14:57:31.830767 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:35659
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:35659
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41855/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/testchainca.cert with env {}
W20260619 14:57:31.927299 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:31.927495 15666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:31.927531 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:31.931376 15666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:31.931423 15666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:31.931438 15666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:31.931483 15666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:31.931514 15666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:57:31.931535 15666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:57:31.931569 15666 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41855/jwks.json
I20260619 14:57:31.936267 15666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:35659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:35659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41855/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:31.936990 15666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:31.937722 15666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:31.944813 15674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.944835 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:31.944839 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:31.945031 15666 server_base.cc:1034] running on GCE node
I20260619 14:57:31.945554 15666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:31.946434 15666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:31.947588 15666 hybrid_clock.cc:648] HybridClock initialized: now 1781881051947572 us; error 25 us; skew 500 ppm
I20260619 14:57:31.949218 15666 webserver.cc:460] Webserver started at http://127.10.71.190:42139/ using document root <none> and password file <none>
I20260619 14:57:31.949715 15666 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:31.949769 15666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:31.949965 15666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:31.951632 15666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "c409cd7da0684b038c3ef12b002ae18d"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "43161ac027728db4cd9b00e87d021209"
server_key_iv: "02906ee01f2c83b6add6a7669cfef576"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.952121 15666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "c409cd7da0684b038c3ef12b002ae18d"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "43161ac027728db4cd9b00e87d021209"
server_key_iv: "02906ee01f2c83b6add6a7669cfef576"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.955257 15666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:57:31.957195 15680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:31.958192 15666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:31.958323 15666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "c409cd7da0684b038c3ef12b002ae18d"
format_stamp: "Formatted at 2026-06-19 14:57:31 on dist-test-slave-3qdh"
server_key: "43161ac027728db4cd9b00e87d021209"
server_key_iv: "02906ee01f2c83b6add6a7669cfef576"
server_key_version: "encryptionkey@0"
I20260619 14:57:31.958415 15666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:31.988833 15666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:31.989434 15666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:31.989609 15666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:31.995584 15666 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:35659
I20260619 14:57:31.995618 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:35659 every 8 connection(s)
I20260619 14:57:31.996428 15666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:31.999054 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:32.003876 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d: Bootstrap starting.
I20260619 14:57:32.005805 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15666
I20260619 14:57:32.005929 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:32.006234 15733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:32.006255 10526 external_mini_cluster.cc:1428] Setting key 693c30ea0d58a79ee7b12ac257283823
I20260619 14:57:32.007031 15733 log.cc:824] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:32.008800 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d: No bootstrap required, opened a new log
I20260619 14:57:32.011864 15733 raft_consensus.cc:348] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c409cd7da0684b038c3ef12b002ae18d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35659 } }
I20260619 14:57:32.012146 15733 raft_consensus.cc:374] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:32.012213 15733 raft_consensus.cc:729] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c409cd7da0684b038c3ef12b002ae18d, State: Initialized, Role: FOLLOWER
I20260619 14:57:32.012707 15733 consensus_queue.cc:260] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c409cd7da0684b038c3ef12b002ae18d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35659 } }
I20260619 14:57:32.012853 15733 raft_consensus.cc:388] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:32.012938 15733 raft_consensus.cc:482] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:32.013072 15733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:32.013903 15733 raft_consensus.cc:504] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c409cd7da0684b038c3ef12b002ae18d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35659 } }
I20260619 14:57:32.014253 15733 leader_election.cc:302] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c409cd7da0684b038c3ef12b002ae18d; no voters:
I20260619 14:57:32.014362 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.007198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.10.71.190:35659)
0619 14:57:32.007747 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.007765 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.007822 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:57:32.008404 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.008420 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.008685 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.008989 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:32.009374 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.010266 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.011079 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.011488 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.011644 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.012510 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:32.012514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:32.012524 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:32.012899 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:32.012980 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:32.013004 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:32.013133 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:32.013452 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 14:57:32.013660 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260619 14:57:32.014577 15733 leader_election.cc:288] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:32.014871 15739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:32.015084 15739 raft_consensus.cc:686] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [term 1 LEADER]: Becoming Leader. State: Replica: c409cd7da0684b038c3ef12b002ae18d, State: Running, Role: LEADER
I20260619 14:57:32.015328 15739 consensus_queue.cc:237] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c409cd7da0684b038c3ef12b002ae18d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35659 } }
I20260619 14:57:32.015599 15733 sys_catalog.cc:564] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:32.016754 15740 sys_catalog.cc:455] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c409cd7da0684b038c3ef12b002ae18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c409cd7da0684b038c3ef12b002ae18d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35659 } } }
I20260619 14:57:32.016878 15740 sys_catalog.cc:458] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.017195 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c409cd7da0684b038c3ef12b002ae18d. Latest consensus state: current_term: 1 leader_uuid: "c409cd7da0684b038c3ef12b002ae18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c409cd7da0684b038c3ef12b002ae18d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35659 } } }
I20260619 14:57:32.017227 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:57:32.017279 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.018405 15747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:32.020987 15747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:32.025718 15747 catalog_manager.cc:1347] Generated new cluster ID: ab1ad053a9984322a77e676acad3ebc2
I20260619 14:57:32.025768 15747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:32.039727 15747 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:32.040669 15747 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:32.046386 15747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c409cd7da0684b038c3ef12b002ae18d: Generated new TSK 0
I20260619 14:57:32.046959 15747 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:57:32.077490 15763 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:57:32.077600 15763 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:57:32.078167 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.062709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.10.71.190:35659)
0619 14:57:32.062815 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.062820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.063041 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:57:32.063270 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.063274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.063327 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.063424 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:57:32.063834 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.064479 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.065371 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.065612 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.065721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.066049 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:57:32.077717 (+ 11668us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:57:32.078003 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 14:57:32.078243 15737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:57:32.079124 15762 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:32.062639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35659 (local address 127.0.0.1:49932)
0619 14:57:32.062946 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:32.062958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:32.063102 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:32.063443 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:32.063446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:32.063454 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:57:32.063563 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.063569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.064618 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.064623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.065253 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.065260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.065730 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.065734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.065819 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.065868 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:57:32.077934 (+ 12066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:57:32.078124 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.71.190:35659: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.71.190:35659: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:57:32.079881 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15666
2026-06-19T14:57:32Z chronyd exiting
I20260619 14:57:32.116164 10526 test_util.cc:175] -----------------------------------------------
I20260619 14:57:32.116214 10526 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781881028756249-10526-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:57:32.117125 10526 mini_oidc.cc:152] Starting JWKS server
I20260619 14:57:32.117909 10526 webserver.cc:460] Webserver started at https://127.0.0.1:38375/ using document root <none> and password file <none>
I20260619 14:57:32.118012 10526 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:57:32.118305 10526 webserver.cc:460] Webserver started at http://127.0.0.1:38441/ using document root <none> and password file <none>
2026-06-19T14:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:32Z Disabled control of system clock
I20260619 14:57:32.126081 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39011
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:39011
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38375/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/testchainca.cert with env {}
W20260619 14:57:32.222630 15769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:32.222818 15769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:32.222851 15769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:32.226444 15769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:32.226481 15769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:32.226495 15769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:32.226508 15769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:32.226521 15769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:57:32.226534 15769 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:57:32.226545 15769 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38375/jwks.json
I20260619 14:57:32.230938 15769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:39011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38375/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:32.231627 15769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:32.232393 15769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:32.239606 15777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:32.239661 15775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:32.239835 15769 server_base.cc:1034] running on GCE node
W20260619 14:57:32.239619 15774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:32.241027 15769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:32.242116 15769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:32.243299 15769 hybrid_clock.cc:648] HybridClock initialized: now 1781881052243277 us; error 36 us; skew 500 ppm
I20260619 14:57:32.244916 15769 webserver.cc:460] Webserver started at http://127.10.71.190:39739/ using document root <none> and password file <none>
I20260619 14:57:32.245383 15769 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:32.245464 15769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:32.245644 15769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:32.247390 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "275cf7a6688c489a99c731532e7a2580"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "bd4ae629ad554c335e5d33a86e122c74"
server_key_iv: "033cb171eaf1c34df4a29d1c634cfdf9"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.247892 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "275cf7a6688c489a99c731532e7a2580"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "bd4ae629ad554c335e5d33a86e122c74"
server_key_iv: "033cb171eaf1c34df4a29d1c634cfdf9"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.251050 15769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:32.253261 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:32.254338 15769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:32.254457 15769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "275cf7a6688c489a99c731532e7a2580"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "bd4ae629ad554c335e5d33a86e122c74"
server_key_iv: "033cb171eaf1c34df4a29d1c634cfdf9"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.254552 15769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:32.274739 15769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:32.275292 15769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:32.275472 15769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:32.281409 15769 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:39011
I20260619 14:57:32.281414 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:39011 every 8 connection(s)
I20260619 14:57:32.282174 15769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:32.284917 15836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:32.289362 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580: Bootstrap starting.
I20260619 14:57:32.290388 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15769
I20260619 14:57:32.290520 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:32.290773 10526 external_mini_cluster.cc:1428] Setting key 9760cc03877f6619747719824438065e
I20260619 14:57:32.291560 15836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:32.292258 15836 log.cc:824] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:32.294083 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580: No bootstrap required, opened a new log
I20260619 14:57:32.297798 15836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275cf7a6688c489a99c731532e7a2580" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39011 } }
I20260619 14:57:32.298038 15836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:32.298108 15836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275cf7a6688c489a99c731532e7a2580, State: Initialized, Role: FOLLOWER
I20260619 14:57:32.298467 15836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275cf7a6688c489a99c731532e7a2580" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39011 } }
I20260619 14:57:32.298568 15836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:32.298647 15836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:32.298741 15836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:32.299093 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.291648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.10.71.190:39011)
0619 14:57:32.292212 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.292227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.292258 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:57:32.292878 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.292892 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.293208 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.293460 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:32.293753 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.294718 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.295917 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.296443 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.296554 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.297327 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:32.297331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:32.297341 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:32.297650 (+ 309us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:32.297697 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:32.297714 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:32.297783 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:32.297986 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 14:57:32.298327 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260619 14:57:32.299495 15836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275cf7a6688c489a99c731532e7a2580" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39011 } }
I20260619 14:57:32.299782 15836 leader_election.cc:302] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 275cf7a6688c489a99c731532e7a2580; no voters:
I20260619 14:57:32.300032 15836 leader_election.cc:288] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:32.300120 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:32.300308 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [term 1 LEADER]: Becoming Leader. State: Replica: 275cf7a6688c489a99c731532e7a2580, State: Running, Role: LEADER
I20260619 14:57:32.300551 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275cf7a6688c489a99c731532e7a2580" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39011 } }
I20260619 14:57:32.301106 15836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:32.301608 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:57:32.301988 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "275cf7a6688c489a99c731532e7a2580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275cf7a6688c489a99c731532e7a2580" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39011 } } }
I20260619 14:57:32.302155 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.302264 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 275cf7a6688c489a99c731532e7a2580. Latest consensus state: current_term: 1 leader_uuid: "275cf7a6688c489a99c731532e7a2580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275cf7a6688c489a99c731532e7a2580" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39011 } } }
I20260619 14:57:32.302340 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.302742 15849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:32.305394 15849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:32.310158 15849 catalog_manager.cc:1347] Generated new cluster ID: e77c9e2131ad433e9b5c2e9333542bc9
I20260619 14:57:32.310210 15849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:32.321497 15849 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:32.322461 15849 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:32.327060 15849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 275cf7a6688c489a99c731532e7a2580: Generated new TSK 0
I20260619 14:57:32.327651 15849 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:57:32.349634 15866 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:57:32.349707 15866 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:57:32.350181 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.335898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.10.71.190:39011)
0619 14:57:32.336006 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.336010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.336184 (+ 174us) server_negotiation.cc:407] Connection header received
0619 14:57:32.336287 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.336290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.336352 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.336434 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:57:32.336670 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.337376 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.338181 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.338486 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.338577 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.338787 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:57:32.349872 (+ 11085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:57:32.350064 (+ 192us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36028: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:57:32.350244 15839 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36028: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 14:57:32.350476 15865 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:32.335830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39011 (local address 127.0.0.1:36028)
0619 14:57:32.336093 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:32.336121 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:32.336198 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:32.336469 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:32.336471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:32.336477 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:57:32.336574 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.336579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.337488 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.337490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.338053 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.338059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.338588 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.338590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.338643 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.338661 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:57:32.350007 (+ 11346us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:57:32.350130 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.71.190:39011: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260619 14:57:32.351130 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15769
2026-06-19T14:57:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (272 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:57:32.389361 10526 mini_oidc.cc:152] Starting JWKS server
I20260619 14:57:32.390458 10526 webserver.cc:460] Webserver started at https://127.0.0.1:46703/ using document root <none> and password file <none>
I20260619 14:57:32.390588 10526 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:57:32.390842 10526 webserver.cc:460] Webserver started at http://127.0.0.1:44923/ using document root <none> and password file <none>
2026-06-19T14:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:32Z Disabled control of system clock
I20260619 14:57:32.398923 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42811
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:42811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46703/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:57:32.496589 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:32.496778 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:32.496809 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:32.500159 15872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:32.500196 15872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:32.500209 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:32.500224 15872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:32.500237 15872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:57:32.500252 15872 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:57:32.500262 15872 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46703/jwks.json
I20260619 14:57:32.504298 15872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:42811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46703/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:32.505019 15872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:32.505995 15872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:32.513054 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:32.513168 15872 server_base.cc:1034] running on GCE node
W20260619 14:57:32.513043 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:32.513054 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:32.513777 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:32.514650 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:32.515867 15872 hybrid_clock.cc:648] HybridClock initialized: now 1781881052515839 us; error 25 us; skew 500 ppm
I20260619 14:57:32.517463 15872 webserver.cc:460] Webserver started at http://127.10.71.190:36545/ using document root <none> and password file <none>
I20260619 14:57:32.517942 15872 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:32.517995 15872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:32.518246 15872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:32.519886 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "128750b565464e4282736882b026196c"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "d96e8a93151030fae3afd113f1c6f396"
server_key_iv: "8ff11bebc60bac6ed39116124d6aaa44"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.520402 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "128750b565464e4282736882b026196c"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "d96e8a93151030fae3afd113f1c6f396"
server_key_iv: "8ff11bebc60bac6ed39116124d6aaa44"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.523350 15872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:57:32.525465 15886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:32.526544 15872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:32.526670 15872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "128750b565464e4282736882b026196c"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "d96e8a93151030fae3afd113f1c6f396"
server_key_iv: "8ff11bebc60bac6ed39116124d6aaa44"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.526764 15872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:32.537180 15872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:32.537714 15872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:32.537910 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:32.544406 15872 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:42811
I20260619 14:57:32.544426 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:42811 every 8 connection(s)
I20260619 14:57:32.545192 15872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:32.547722 15939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:32.552685 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c: Bootstrap starting.
I20260619 14:57:32.553257 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15872
I20260619 14:57:32.553398 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:32.553682 10526 external_mini_cluster.cc:1428] Setting key f344a0b93f3a1ad0c985fb39dbecd9bc
I20260619 14:57:32.555225 15939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:32.555891 15939 log.cc:824] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:32.557469 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c: No bootstrap required, opened a new log
I20260619 14:57:32.560659 15939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128750b565464e4282736882b026196c" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42811 } }
I20260619 14:57:32.560832 15939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:32.560883 15939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128750b565464e4282736882b026196c, State: Initialized, Role: FOLLOWER
I20260619 14:57:32.561224 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.554754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.10.71.190:42811)
0619 14:57:32.555284 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.555301 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.555338 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:57:32.555874 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.555887 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.556232 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.556537 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:32.556874 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.557692 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.558494 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.558900 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.559055 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.559819 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:32.559823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:32.559833 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:32.560125 (+ 292us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:32.560166 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:32.560182 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:32.560275 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:32.560398 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:57:32.560610 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260619 14:57:32.561277 15939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128750b565464e4282736882b026196c" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42811 } }
I20260619 14:57:32.561426 15939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:32.561478 15939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:32.561558 15939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:32.562359 15939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128750b565464e4282736882b026196c" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42811 } }
I20260619 14:57:32.562692 15939 leader_election.cc:302] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128750b565464e4282736882b026196c; no voters:
I20260619 14:57:32.563009 15939 leader_election.cc:288] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:32.563314 15944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:32.563522 15944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [term 1 LEADER]: Becoming Leader. State: Replica: 128750b565464e4282736882b026196c, State: Running, Role: LEADER
I20260619 14:57:32.563808 15944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128750b565464e4282736882b026196c" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42811 } }
I20260619 14:57:32.563979 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:57:32.564426 15939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:32.565065 15945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128750b565464e4282736882b026196c. Latest consensus state: current_term: 1 leader_uuid: "128750b565464e4282736882b026196c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128750b565464e4282736882b026196c" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42811 } } }
I20260619 14:57:32.565234 15945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.565348 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128750b565464e4282736882b026196c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128750b565464e4282736882b026196c" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42811 } } }
I20260619 14:57:32.565459 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.565690 15951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:32.568199 15951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:32.577718 15951 catalog_manager.cc:1347] Generated new cluster ID: b4251ddafa144aa59980936e1c0869a3
I20260619 14:57:32.577787 15951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:32.589797 15951 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:32.590585 15951 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:32.600934 15951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128750b565464e4282736882b026196c: Generated new TSK 0
I20260619 14:57:32.601498 15951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:57:32.613606 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.609124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.10.71.190:42811)
0619 14:57:32.609241 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.609245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.609368 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:57:32.609508 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.609510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.609548 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.609619 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:57:32.609862 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.610409 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.611483 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.611666 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.611728 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.612020 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:57:32.613282 (+ 1262us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46703/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:57:32.613483 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46703/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 14:57:32.613657 15942 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46703/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:57:32.615331 15968 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:32.609006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42811 (local address 127.0.0.1:45602)
0619 14:57:32.609284 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:32.609298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:32.609396 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:32.609661 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:32.609664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:32.609671 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:57:32.609775 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.609780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.610509 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.610512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.611374 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.611385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.611764 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.611768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.611844 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.611884 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:57:32.613430 (+ 1546us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46703/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:57:32.615186 (+ 1756us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.71.190:42811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46703/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1575,"thread_start_us":90,"threads_started":1}
I20260619 14:57:32.616037 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15872
E20260619 14:57:32.625672 15969 webserver.cc:533] Webserver: sslize failed:
E20260619 14:57:32.625759 15969 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T14:57:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (265 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:32Z Disabled control of system clock
I20260619 14:57:32.662477 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:44371
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:44371
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/testchainca.cert with env {}
W20260619 14:57:32.760224 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:32.760488 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:32.760525 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:32.764400 15973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:32.764444 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:32.764460 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:32.764477 15973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:32.764492 15973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:57:32.764508 15973 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:57:32.764528 15973 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:57:32.769106 15973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:44371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:44371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:32.769860 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:32.770628 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:32.778041 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:32.778127 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:32.778088 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:32.778227 15973 server_base.cc:1034] running on GCE node
I20260619 14:57:32.778702 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:32.779632 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:32.780850 15973 hybrid_clock.cc:648] HybridClock initialized: now 1781881052780833 us; error 43 us; skew 500 ppm
I20260619 14:57:32.782402 15973 webserver.cc:460] Webserver started at http://127.10.71.190:44311/ using document root <none> and password file <none>
I20260619 14:57:32.782861 15973 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:32.782940 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:32.783191 15973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:32.784957 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "eeb4d48bc5d947db9b767e4b40d43f80"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "c9b1ab0fad4c4f4cfc92456aa1412490"
server_key_iv: "c14e74ddc96f458c3b1c89fdef836cf7"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.785452 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "eeb4d48bc5d947db9b767e4b40d43f80"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "c9b1ab0fad4c4f4cfc92456aa1412490"
server_key_iv: "c14e74ddc96f458c3b1c89fdef836cf7"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.788609 15973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:32.790704 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:32.791782 15973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:32.791918 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "eeb4d48bc5d947db9b767e4b40d43f80"
format_stamp: "Formatted at 2026-06-19 14:57:32 on dist-test-slave-3qdh"
server_key: "c9b1ab0fad4c4f4cfc92456aa1412490"
server_key_iv: "c14e74ddc96f458c3b1c89fdef836cf7"
server_key_version: "encryptionkey@0"
I20260619 14:57:32.792016 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:32.801314 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:32.801791 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:32.801954 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:32.807761 15973 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:44371
I20260619 14:57:32.807809 16039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:44371 every 8 connection(s)
I20260619 14:57:32.808600 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:32.811134 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:32.816951 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 15973
I20260619 14:57:32.817051 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:32.817365 10526 external_mini_cluster.cc:1428] Setting key e39b812587666566d6b86f408b6b0eba
I20260619 14:57:32.817749 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80: Bootstrap starting.
I20260619 14:57:32.820331 16040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:32.821134 16040 log.cc:824] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:32.822937 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80: No bootstrap required, opened a new log
I20260619 14:57:32.827066 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.818862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.10.71.190:44371)
0619 14:57:32.819479 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.819496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.819533 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:57:32.820356 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.820379 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.820835 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.821161 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:32.821556 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.822766 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.823666 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.824378 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.824520 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.825185 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:32.825191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:32.825205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:57:32.825642 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:32.825703 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:57:32.825726 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:32.825835 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:32.825946 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 14:57:32.826231 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260619 14:57:32.827059 16040 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44371 } }
I20260619 14:57:32.827301 16040 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:32.827358 16040 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb4d48bc5d947db9b767e4b40d43f80, State: Initialized, Role: FOLLOWER
I20260619 14:57:32.827734 16040 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44371 } }
I20260619 14:57:32.827913 16040 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:32.827998 16040 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:32.828135 16040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:32.829103 16040 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44371 } }
I20260619 14:57:32.829430 16040 leader_election.cc:302] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeb4d48bc5d947db9b767e4b40d43f80; no voters:
I20260619 14:57:32.829728 16040 leader_election.cc:288] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:32.829823 16045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:32.830024 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:57:32.830019 16045 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [term 1 LEADER]: Becoming Leader. State: Replica: eeb4d48bc5d947db9b767e4b40d43f80, State: Running, Role: LEADER
I20260619 14:57:32.830278 16045 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44371 } }
I20260619 14:57:32.830782 16040 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:32.831501 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb4d48bc5d947db9b767e4b40d43f80. Latest consensus state: current_term: 1 leader_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44371 } } }
I20260619 14:57:32.831605 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.831646 16046 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb4d48bc5d947db9b767e4b40d43f80" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 44371 } } }
I20260619 14:57:32.831753 16046 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:32.831943 16052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:32.834617 16052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:32.839143 16052 catalog_manager.cc:1347] Generated new cluster ID: 040201f874e544b3882d636666dcbbc5
I20260619 14:57:32.839197 16052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:32.860338 16052 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:32.861310 16052 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:32.869510 16052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeb4d48bc5d947db9b767e4b40d43f80: Generated new TSK 0
I20260619 14:57:32.870067 16052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:57:32.893927 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:32.875728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.10.71.190:44371)
0619 14:57:32.875835 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:32.875841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:32.876016 (+ 175us) server_negotiation.cc:407] Connection header received
0619 14:57:32.876195 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:32.876197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:32.876247 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:32.876363 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:57:32.876588 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.877026 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.878154 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.878340 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.878406 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.878713 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:57:32.893678 (+ 14965us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:57:32.893845 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:57:32.893996 16043 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 14:57:32.894416 16069 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:32.875651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:44371 (local address 127.0.0.1:40364)
0619 14:57:32.875940 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:32.875955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:32.876075 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:32.876361 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:32.876364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:32.876372 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:57:32.876484 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.876491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.877127 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.877131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.878039 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:32.878055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:32.878451 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:32.878455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:32.878555 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:32.878581 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:57:32.893829 (+ 15248us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:57:32.893990 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.71.190:44371: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260619 14:57:32.894971 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 15973
2026-06-19T14:57:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (257 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (109 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (105 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:33 dist-test-slave-3qdh krb5kdc[16081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:33 dist-test-slave-3qdh krb5kdc[16081](info): set up 2 sockets
Jun 19 14:57:33 dist-test-slave-3qdh krb5kdc[16081](info): commencing operation
krb5kdc: starting...
W20260619 14:57:34.060139 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:34.168148 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40577
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:34.262518 16094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:34.262699 16094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:34.262730 16094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:34.266368 16094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:34.266407 16094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:34.266423 16094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:34.266438 16094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:34.266449 16094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:34.270771 16094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:34.271487 16094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:34.272205 16094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:34.279438 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:34.279412 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:34.279464 16099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.279558 16094 server_base.cc:1034] running on GCE node
I20260619 14:57:34.280000 16094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:34.281018 16094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:34.282169 16094 hybrid_clock.cc:648] HybridClock initialized: now 1781881054282144 us; error 37 us; skew 500 ppm
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.284724 16094 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:34.285496 16094 webserver.cc:460] Webserver started at http://127.10.71.190:43993/ using document root <none> and password file <none>
I20260619 14:57:34.285981 16094 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:34.286067 16094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:34.286273 16094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:34.288000 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "a2ed458f34c0427084049dbf20433dee"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "7ec99855e9d05c0085d2636cc5def050"
server_key_iv: "d7ec36324c7710de997741767cb47981"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.288519 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "a2ed458f34c0427084049dbf20433dee"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "7ec99855e9d05c0085d2636cc5def050"
server_key_iv: "d7ec36324c7710de997741767cb47981"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.291637 16094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:57:34.293718 16109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.294775 16094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:34.294898 16094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "a2ed458f34c0427084049dbf20433dee"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "7ec99855e9d05c0085d2636cc5def050"
server_key_iv: "d7ec36324c7710de997741767cb47981"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.294991 16094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:34.306838 16094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:34.309495 16094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:34.309667 16094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:34.315405 16094 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:40577
I20260619 14:57:34.315465 16161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:40577 every 8 connection(s)
I20260619 14:57:34.316145 16094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:34.318317 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:34.322510 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16094
I20260619 14:57:34.322652 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:34.322911 10526 external_mini_cluster.cc:1428] Setting key 54e3b27fc3fa762aaff84946eff4da7a
I20260619 14:57:34.322969 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee: Bootstrap starting.
I20260619 14:57:34.325173 16162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:34.325930 16162 log.cc:824] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:34.327628 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee: No bootstrap required, opened a new log
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:34.331459 16162 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed458f34c0427084049dbf20433dee" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40577 } }
I20260619 14:57:34.331622 16162 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:34.331673 16162 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ed458f34c0427084049dbf20433dee, State: Initialized, Role: FOLLOWER
I20260619 14:57:34.332048 16162 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed458f34c0427084049dbf20433dee" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40577 } }
I20260619 14:57:34.332187 16162 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:34.332259 16162 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:34.332382 16162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:34.333093 16162 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed458f34c0427084049dbf20433dee" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40577 } }
I20260619 14:57:34.333421 16162 leader_election.cc:302] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ed458f34c0427084049dbf20433dee; no voters:
I20260619 14:57:34.333676 16162 leader_election.cc:288] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:34.333778 16167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:34.334004 16167 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [term 1 LEADER]: Becoming Leader. State: Replica: a2ed458f34c0427084049dbf20433dee, State: Running, Role: LEADER
I20260619 14:57:34.334343 16167 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed458f34c0427084049dbf20433dee" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40577 } }
I20260619 14:57:34.334676 16162 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:34.335383 16169 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ed458f34c0427084049dbf20433dee. Latest consensus state: current_term: 1 leader_uuid: "a2ed458f34c0427084049dbf20433dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed458f34c0427084049dbf20433dee" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40577 } } }
I20260619 14:57:34.335526 16169 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:34.335919 16176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:34.336197 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.324038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.10.71.190:40577)
0619 14:57:34.324442 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.324455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.324485 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:57:34.324983 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.324995 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.325278 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.325508 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.326294 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.327237 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.328394 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.328809 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.328915 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.331320 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:34.331325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:34.331335 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:34.331357 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:34.332993 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.333497 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.333502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.333514 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.333678 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.334011 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.334018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.334022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.334466 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:34.334650 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:34.335082 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 14:57:34.335332 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260619 14:57:34.336901 16168 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ed458f34c0427084049dbf20433dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed458f34c0427084049dbf20433dee" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40577 } } }
I20260619 14:57:34.337040 16168 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:34.338583 16176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:34.343050 16176 catalog_manager.cc:1347] Generated new cluster ID: 7a172a2cd84d404aa53667aa6fc1898b
I20260619 14:57:34.343102 16176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:34.361251 16176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:34.362274 16176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:34.369990 16176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ed458f34c0427084049dbf20433dee: Generated new TSK 0
I20260619 14:57:34.370554 16176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:34.395150 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40577
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:34.491894 16188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:34.492121 16188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:34.492162 16188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:34.495950 16188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:34.495998 16188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:34.496071 16188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:34.500741 16188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:34.501571 16188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:34.502400 16188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:34.510624 16194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:34.510609 16196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.510649 16188 server_base.cc:1034] running on GCE node
W20260619 14:57:34.510609 16193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.511021 16188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:34.511505 16188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:34.512662 16188 hybrid_clock.cc:648] HybridClock initialized: now 1781881054512640 us; error 33 us; skew 500 ppm
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.515144 16188 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:34.515807 16188 webserver.cc:460] Webserver started at http://127.10.71.129:35355/ using document root <none> and password file <none>
I20260619 14:57:34.516243 16188 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:34.516330 16188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:34.516521 16188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:34.518123 16188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "92f913c974db4d94878a0b8151b18283"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "1ea88c1c5ff6a41c2948e357366904bc"
server_key_iv: "f461fdfcc13f503339f51f9298f0334b"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.518615 16188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "92f913c974db4d94878a0b8151b18283"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "1ea88c1c5ff6a41c2948e357366904bc"
server_key_iv: "f461fdfcc13f503339f51f9298f0334b"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.521757 16188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:57:34.523795 16203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.524937 16188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:34.525054 16188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "92f913c974db4d94878a0b8151b18283"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "1ea88c1c5ff6a41c2948e357366904bc"
server_key_iv: "f461fdfcc13f503339f51f9298f0334b"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.525163 16188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:34.555718 16188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:34.558235 16188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:34.558400 16188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:34.558864 16188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:34.559511 16188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:34.559579 16188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.559654 16188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:34.559698 16188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.568025 16188 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:44687
I20260619 14:57:34.568073 16316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:44687 every 8 connection(s)
I20260619 14:57:34.568892 16188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:34.570180 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16188
I20260619 14:57:34.570290 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:34.570592 10526 external_mini_cluster.cc:1428] Setting key 3482a63675dc8e360362c97d1c432e96
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:34.581995 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.570451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:56883 (local address 127.10.71.190:40577)
0619 14:57:34.570591 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.570595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.571447 (+ 852us) server_negotiation.cc:407] Connection header received
0619 14:57:34.573065 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.573068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.573108 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.573172 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.574406 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.575115 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.576184 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.576598 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.576714 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.578880 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:34.578885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:34.578891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:34.578923 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:34.580170 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.580648 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.580650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.580651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.580717 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.580974 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.580976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.580977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.581127 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:34.581198 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:34.581651 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 14:57:34.581788 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:34.582809 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.570807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40577 (local address 127.10.71.129:56883)
0619 14:57:34.571295 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:34.571342 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:34.572868 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:34.573363 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:34.573373 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:34.573823 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:34.574239 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:34.574252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.575256 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.575259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:34.575996 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:34.576005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.576729 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.576732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:34.576858 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.577371 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:34.577385 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:34.578722 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:34.580278 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:34.580295 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:34.580319 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:34.580552 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:34.580790 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:34.580792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:34.580793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:34.580902 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:34.581225 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:34.581229 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:34.581420 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 14:57:34.581605 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 14:57:34.581874 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":122,"threads_started":1}
I20260619 14:57:34.583967 16317 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40577
I20260619 14:57:34.584185 16317 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:34.584636 16317 heartbeater.cc:511] Master 127.10.71.190:40577 requested a full tablet report, sending...
I20260619 14:57:34.586074 16126 ts_manager.cc:194] Registered new tserver with Master: 92f913c974db4d94878a0b8151b18283 (127.10.71.129:44687)
I20260619 14:57:34.587550 16126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:56883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:34.595101 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40577
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:34.689030 16322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:34.689220 16322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:34.689252 16322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:34.692667 16322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:34.692704 16322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:34.692750 16322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:34.697250 16322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:34.697953 16322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:34.698709 16322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:34.706668 16327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:34.706669 16330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.706967 16322 server_base.cc:1034] running on GCE node
W20260619 14:57:34.707109 16328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.707610 16322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:34.708127 16322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:34.709255 16322 hybrid_clock.cc:648] HybridClock initialized: now 1781881054709261 us; error 44 us; skew 500 ppm
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.711437 16322 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:34.712119 16322 webserver.cc:460] Webserver started at http://127.10.71.130:37661/ using document root <none> and password file <none>
I20260619 14:57:34.712612 16322 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:34.712666 16322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:34.712850 16322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:34.714610 16322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "7fae76bc66714a428801eb8336bef3a3"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "f7703ef3b373adbc10458581d6762ad9"
server_key_iv: "92d4ecf70428acf0fd1e709d794ec097"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.715071 16322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "7fae76bc66714a428801eb8336bef3a3"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "f7703ef3b373adbc10458581d6762ad9"
server_key_iv: "92d4ecf70428acf0fd1e709d794ec097"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.718053 16322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:57:34.719967 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.720939 16322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:34.721053 16322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "7fae76bc66714a428801eb8336bef3a3"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "f7703ef3b373adbc10458581d6762ad9"
server_key_iv: "92d4ecf70428acf0fd1e709d794ec097"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.721190 16322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:34.738920 16322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:34.741894 16322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:34.742022 16322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:34.742449 16322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:34.743088 16322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:34.743136 16322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.743175 16322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:34.743196 16322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.750923 16322 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:46277
I20260619 14:57:34.750988 16450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:46277 every 8 connection(s)
I20260619 14:57:34.751648 16322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:34.759927 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16322
I20260619 14:57:34.760079 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:34.760401 10526 external_mini_cluster.cc:1428] Setting key dd5a14d9995987963a6fafabfc5c00f3
I20260619 14:57:34.763370 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.753174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:53389 (local address 127.10.71.190:40577)
0619 14:57:34.753310 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.753314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.753945 (+ 631us) server_negotiation.cc:407] Connection header received
0619 14:57:34.755167 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.755170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.755224 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.755314 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.756250 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.756749 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.757789 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.757980 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.758075 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.759962 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:34.759965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:34.759966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:34.760004 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:34.761481 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.761952 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.761955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.761957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.762038 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.762307 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.762312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.762316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.762527 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:34.762623 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:34.763008 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:57:34.763125 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:57:34.764110 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.753417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40577 (local address 127.10.71.130:53389)
0619 14:57:34.753810 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:34.753845 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:34.754973 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:34.755461 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:34.755467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:34.755743 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:34.756128 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:34.756139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.756855 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.756858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:34.757678 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:34.757686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.758107 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.758111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:34.758250 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.758631 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:34.758643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:34.759816 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:34.761589 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:34.761593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:34.761604 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:34.761866 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:34.762118 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:34.762120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:34.762121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:34.762225 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:34.762625 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:34.762629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:34.762796 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 14:57:34.762938 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:57:34.763157 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260619 14:57:34.765093 16451 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40577
I20260619 14:57:34.765280 16451 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:34.765743 16451 heartbeater.cc:511] Master 127.10.71.190:40577 requested a full tablet report, sending...
I20260619 14:57:34.766736 16126 ts_manager.cc:194] Registered new tserver with Master: 7fae76bc66714a428801eb8336bef3a3 (127.10.71.130:46277)
I20260619 14:57:34.767534 16126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:53389
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:34.784881 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40577
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:34.882568 16456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:34.882787 16456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:34.882848 16456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:34.886497 16456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:34.886564 16456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:34.886636 16456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:34.891077 16456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:34.891824 16456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:34.892647 16456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:34.900466 16464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:34.900487 16461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.900548 16456 server_base.cc:1034] running on GCE node
W20260619 14:57:34.900481 16462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:34.900981 16456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:34.901515 16456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:34.902700 16456 hybrid_clock.cc:648] HybridClock initialized: now 1781881054902686 us; error 28 us; skew 500 ppm
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.905165 16456 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:34.905903 16456 webserver.cc:460] Webserver started at http://127.10.71.131:35991/ using document root <none> and password file <none>
I20260619 14:57:34.906301 16456 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:34.906349 16456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:34.906476 16456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:34.908057 16456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "3ab820bb10004ace99d95e843c89e69e"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "bd4babc59425b9dbd7aa19eb12c50eea"
server_key_iv: "fbc636a91d09488beff230c191e42032"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.908540 16456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "3ab820bb10004ace99d95e843c89e69e"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "bd4babc59425b9dbd7aa19eb12c50eea"
server_key_iv: "fbc636a91d09488beff230c191e42032"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.911365 16456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:57:34.913411 16471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.914403 16456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:34.914574 16456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "3ab820bb10004ace99d95e843c89e69e"
format_stamp: "Formatted at 2026-06-19 14:57:34 on dist-test-slave-3qdh"
server_key: "bd4babc59425b9dbd7aa19eb12c50eea"
server_key_iv: "fbc636a91d09488beff230c191e42032"
server_key_version: "encryptionkey@0"
I20260619 14:57:34.914675 16456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:34.926047 16456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:34.928949 16456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:34.929117 16456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:34.929576 16456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:34.930234 16456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:34.930297 16456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.930356 16456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:34.930384 16456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:34.938853 16456 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:34593
I20260619 14:57:34.938886 16584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:34593 every 8 connection(s)
I20260619 14:57:34.939568 16456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:34.949254 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16456
I20260619 14:57:34.949369 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:34.949653 10526 external_mini_cluster.cc:1428] Setting key 976181efbe0f93f1fd8033c138ef24c0
I20260619 14:57:34.951460 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.941131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:35925 (local address 127.10.71.190:40577)
0619 14:57:34.941243 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.941247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.941875 (+ 628us) server_negotiation.cc:407] Connection header received
0619 14:57:34.942940 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.942943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.942993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.943060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.944052 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.944667 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.945702 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.945902 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.946001 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.948084 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:34.948087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:34.948089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:34.948110 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:34.949353 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.949826 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.949829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.949831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.949915 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.950307 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.950310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.950312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.950554 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:34.950651 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:34.951212 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 14:57:34.951357 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:34.952072 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.941393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40577 (local address 127.10.71.131:35925)
0619 14:57:34.941751 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:34.941783 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:34.942769 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:34.943217 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:34.943222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:34.943535 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:34.943911 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:34.943922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.944779 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.944783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:34.945577 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:34.945585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.946028 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.946031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:34.946181 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.946612 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:34.946625 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:34.947929 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:34.949472 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:34.949478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:34.949490 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:34.949710 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:34.950023 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:34.950025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:34.950027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:34.950195 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:34.950661 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:34.950667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:34.950888 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:57:34.951092 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 14:57:34.951359 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260619 14:57:34.953001 16585 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40577
I20260619 14:57:34.953212 16585 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:34.953616 16585 heartbeater.cc:511] Master 127.10.71.190:40577 requested a full tablet report, sending...
I20260619 14:57:34.954547 16126 ts_manager.cc:194] Registered new tserver with Master: 3ab820bb10004ace99d95e843c89e69e (127.10.71.131:34593)
I20260619 14:57:34.955046 16126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:35925
I20260619 14:57:34.963131 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/test-admin.keytab.
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.976655 10526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 14:57:34.979111 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:34.977799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.10.71.129:44687)
0619 14:57:34.978100 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.978109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.978130 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:57:34.978185 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.978189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.978342 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.978448 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.978874 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49054: BlockingRecv error: recv got EOF from 127.0.0.1:49054 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
W20260619 14:57:34.979118 10526 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.979964 10526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.980849 10526 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:34.989532 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.981617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.10.71.129:44687)
0619 14:57:34.981728 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.981733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.981790 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:57:34.981935 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.981937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.981970 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.982046 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.982525 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.983330 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.984172 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.984593 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.984705 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.986678 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:34.986685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:34.986695 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:34.986714 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:34.987917 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.988364 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.988369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.988370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.988442 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:34.988713 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:34.988718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:34.988720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:34.988946 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:34.989136 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:34.989300 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 14:57:34.989424 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:57:34.992676 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:34.991740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.10.71.129:44687)
0619 14:57:34.991844 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.991847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.991952 (+ 105us) server_negotiation.cc:407] Connection header received
0619 14:57:34.992081 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.992085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.992124 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.992196 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.992593 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49058: BlockingRecv error: recv got EOF from 127.0.0.1:49058 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:57:34.992980 10526 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc)
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.993984 10526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:34.994825 10526 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:57:34 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:35.002586 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:34.995451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.10.71.129:44687)
0619 14:57:34.995552 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:34.995555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:34.995690 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:57:34.995769 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:34.995771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:34.995798 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:34.995855 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:34.996383 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.996857 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.997588 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:34.997783 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:34.997857 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:34.999946 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:34.999948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:34.999950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:34.999968 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:35.001191 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:35.001558 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:35.001563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:35.001564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:35.001634 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:35.001881 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:35.001885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:35.001891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:35.002030 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:35.002162 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:35.002373 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:57:35.002470 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:57:35.004884 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:35.004046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.10.71.129:44687)
0619 14:57:35.004186 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:35.004190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:35.004203 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:35.004333 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:35.004337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:35.004384 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:35.004459 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:35.004785 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49078: BlockingRecv error: recv got EOF from 127.0.0.1:49078 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:57:35.004953 10526 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781881028756249-10526-0/krb5kdc/krb5cc)
Jun 19 14:57:35 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:35.005788 10526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:57:35 dist-test-slave-3qdh krb5kdc[16081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:35.006529 10526 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:57:35 dist-test-slave-3qdh krb5kdc[16081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 14:57:35.013509 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:35.007180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.10.71.129:44687)
0619 14:57:35.007264 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:35.007266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:35.007418 (+ 152us) server_negotiation.cc:407] Connection header received
0619 14:57:35.007474 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:35.007476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:35.007505 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:35.007569 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:35.007961 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:35.008436 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:35.009090 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:35.009257 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:35.009322 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:35.011020 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:35.011023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:35.011024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:35.011041 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:35.012272 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:35.012645 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:35.012648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:35.012649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:35.012710 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:35.012924 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:35.012926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:35.012927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:35.013063 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:35.013162 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:35.013345 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:57:35.013421 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:57:35.014374 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16188
I20260619 14:57:35.019590 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16322
I20260619 14:57:35.024459 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16456
I20260619 14:57:35.029152 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16094
2026-06-19T14:57:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1922 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:35 dist-test-slave-3qdh krb5kdc[16605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:35 dist-test-slave-3qdh krb5kdc[16605](info): set up 2 sockets
Jun 19 14:57:35 dist-test-slave-3qdh krb5kdc[16605](info): commencing operation
krb5kdc: starting...
W20260619 14:57:35.980189 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab.
Entry for principal oryx/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab.
I20260619 14:57:36.085357 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:36521
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab
--principal=oryx/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:36.180519 16618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:36.180707 16618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:36.180771 16618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:36.184625 16618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:36.184666 16618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:36.184684 16618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:36.184732 16618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:36.184759 16618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:36.189052 16618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab
--principal=oryx/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:36.189996 16618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:36.190858 16618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:36.198249 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:36.198324 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.198494 16618 server_base.cc:1034] running on GCE node
W20260619 14:57:36.198709 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.199188 16618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:36.200040 16618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:36.201189 16618 hybrid_clock.cc:648] HybridClock initialized: now 1781881056201171 us; error 29 us; skew 500 ppm
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:36.207540 16618 init.cc:377] Logged in from keytab as oryx/127.10.71.190@KRBTEST.COM (short username oryx)
I20260619 14:57:36.208249 16618 webserver.cc:460] Webserver started at http://127.10.71.190:44939/ using document root <none> and password file <none>
I20260619 14:57:36.208720 16618 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:36.208774 16618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:36.208994 16618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:36.210544 16618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "359478d2a1624cadbfdc4d89d8997014"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "8c64a89dcff5d769cdce0ff58e07c340"
server_key_iv: "0f5c65029cc95990e3176a5a2be7243e"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.210951 16618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "359478d2a1624cadbfdc4d89d8997014"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "8c64a89dcff5d769cdce0ff58e07c340"
server_key_iv: "0f5c65029cc95990e3176a5a2be7243e"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.214043 16618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:36.216042 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.217092 16618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:36.217209 16618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "359478d2a1624cadbfdc4d89d8997014"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "8c64a89dcff5d769cdce0ff58e07c340"
server_key_iv: "0f5c65029cc95990e3176a5a2be7243e"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.217293 16618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:36.240020 16618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:36.242228 16618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:36.242393 16618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:36.248289 16618 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:36521
I20260619 14:57:36.248546 16685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:36521 every 8 connection(s)
I20260619 14:57:36.249147 16618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:36.249747 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16618
I20260619 14:57:36.249874 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:36.250130 10526 external_mini_cluster.cc:1428] Setting key a64e82b7e5dffd43e7e425dfa42de96a
I20260619 14:57:36.251771 16686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:36.257853 16686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Bootstrap starting.
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:36.260128 16686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:36.260895 16686 log.cc:824] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:36.262749 16686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: No bootstrap required, opened a new log
I20260619 14:57:36.263705 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.251211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.10.71.190:36521)
0619 14:57:36.251550 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:36.251563 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:36.251592 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:57:36.252164 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:36.252179 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:36.252508 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:36.252745 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:36.253577 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.254913 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.255963 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.256610 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.256756 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.259323 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:36.259330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:36.259343 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:36.259370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:36.261116 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.261584 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.261589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.261594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.261702 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.261928 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.261930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.261932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.262319 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:36.262471 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:36.262670 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 14:57:36.262909 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260619 14:57:36.266093 16686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } }
I20260619 14:57:36.266279 16686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:36.266312 16686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359478d2a1624cadbfdc4d89d8997014, State: Initialized, Role: FOLLOWER
I20260619 14:57:36.266664 16686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } }
I20260619 14:57:36.266770 16686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:36.266813 16686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:36.266888 16686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:36.267534 16686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } }
I20260619 14:57:36.267773 16686 leader_election.cc:302] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 359478d2a1624cadbfdc4d89d8997014; no voters:
I20260619 14:57:36.267977 16686 leader_election.cc:288] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:36.268097 16691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:36.268334 16691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 LEADER]: Becoming Leader. State: Replica: 359478d2a1624cadbfdc4d89d8997014, State: Running, Role: LEADER
I20260619 14:57:36.268585 16691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } }
I20260619 14:57:36.269014 16686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:36.270030 16693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359478d2a1624cadbfdc4d89d8997014. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } }
I20260619 14:57:36.270143 16693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:36.270465 16700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:36.270423 16692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } }
I20260619 14:57:36.270923 16692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:36.273110 16700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:36.277937 16700 catalog_manager.cc:1347] Generated new cluster ID: 262ce88f198349a4aae271a44ca707fc
I20260619 14:57:36.278025 16700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:36.300621 16700 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:36.301354 16700 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:36.311749 16700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Generated new TSK 0
I20260619 14:57:36.312256 16700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab.
Entry for principal oryx/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab.
I20260619 14:57:36.340663 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36521
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:36.440102 16712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:36.440353 16712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:36.440403 16712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:36.443939 16712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:36.443980 16712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:36.444036 16712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:36.448861 16712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:36.449582 16712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:36.450323 16712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:36.458648 16717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.458707 16712 server_base.cc:1034] running on GCE node
W20260619 14:57:36.458696 16720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:36.458648 16718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.459249 16712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:36.459792 16712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:36.460959 16712 hybrid_clock.cc:648] HybridClock initialized: now 1781881056460948 us; error 32 us; skew 500 ppm
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:36.463373 16712 init.cc:377] Logged in from keytab as oryx/127.10.71.129@KRBTEST.COM (short username oryx)
I20260619 14:57:36.464128 16712 webserver.cc:460] Webserver started at http://127.10.71.129:44215/ using document root <none> and password file <none>
I20260619 14:57:36.464687 16712 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:36.464767 16712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:36.464962 16712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:36.466562 16712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "65529b9919b04d4b923c9b3c52f0855d"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "312abf441d1b9070cc640ec77afb8cf8"
server_key_iv: "ec874e74052735cd33721780e00a1cc0"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.467039 16712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "65529b9919b04d4b923c9b3c52f0855d"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "312abf441d1b9070cc640ec77afb8cf8"
server_key_iv: "ec874e74052735cd33721780e00a1cc0"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.470214 16712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:36.472165 16727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.473192 16712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:36.473307 16712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "65529b9919b04d4b923c9b3c52f0855d"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "312abf441d1b9070cc640ec77afb8cf8"
server_key_iv: "ec874e74052735cd33721780e00a1cc0"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.473426 16712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:36.482753 16712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:36.485308 16712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:36.485476 16712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:36.485937 16712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:36.486588 16712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:36.486629 16712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.486686 16712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:36.486714 16712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.493932 16712 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:46755
I20260619 14:57:36.493945 16840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:46755 every 8 connection(s)
I20260619 14:57:36.494692 16712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:36.495256 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16712
I20260619 14:57:36.495376 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:36.495623 10526 external_mini_cluster.cc:1428] Setting key 1b00956e3731ba5ae64e24ed50d1a6d2
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:36.507783 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.496221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:58223 (local address 127.10.71.190:36521)
0619 14:57:36.496385 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:36.496389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:36.497195 (+ 806us) server_negotiation.cc:407] Connection header received
0619 14:57:36.499058 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:36.499062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:36.499110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:36.499181 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:36.500102 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.500919 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.501892 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.502324 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.502442 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.504615 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:36.504620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:36.504623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:36.504650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:36.505859 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.506373 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.506376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.506377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.506442 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.506711 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.506713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.506714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.506869 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:36.506953 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:36.507427 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 14:57:36.507551 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:57:36.508656 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.496604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:58223)
0619 14:57:36.497068 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:36.497116 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:36.498870 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:36.499324 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:36.499330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:36.499618 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:36.499970 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:36.499980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.501035 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.501038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:36.501779 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:36.501788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.502447 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.502450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:36.502564 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.503045 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:36.503059 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:36.504429 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:36.505947 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:36.505951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:36.505959 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:36.506296 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:36.506510 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:36.506512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:36.506513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:36.506622 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:36.507006 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:36.507014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:36.507246 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 14:57:36.507434 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:57:36.507696 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260619 14:57:36.509716 16841 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36521
I20260619 14:57:36.509908 16841 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:36.510289 16841 heartbeater.cc:511] Master 127.10.71.190:36521 requested a full tablet report, sending...
I20260619 14:57:36.511510 16650 ts_manager.cc:194] Registered new tserver with Master: 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755)
I20260619 14:57:36.512496 16650 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:58223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab.
Entry for principal oryx/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab.
I20260619 14:57:36.520475 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36521
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:36.615123 16846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:36.615337 16846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:36.615401 16846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:36.618695 16846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:36.618762 16846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:36.618831 16846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:36.623111 16846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:36.623906 16846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:36.624763 16846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:36.632503 16851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.632571 16846 server_base.cc:1034] running on GCE node
W20260619 14:57:36.632509 16854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:36.632511 16852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.632987 16846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:36.633494 16846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:36.634670 16846 hybrid_clock.cc:648] HybridClock initialized: now 1781881056634660 us; error 33 us; skew 500 ppm
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:36.636862 16846 init.cc:377] Logged in from keytab as oryx/127.10.71.130@KRBTEST.COM (short username oryx)
I20260619 14:57:36.637509 16846 webserver.cc:460] Webserver started at http://127.10.71.130:46151/ using document root <none> and password file <none>
I20260619 14:57:36.637969 16846 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:36.638051 16846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:36.638243 16846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:36.639853 16846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "6eeb517267474aafaba6681953067269"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "6f64a829bad175ccdc9ac07e0ae8c69c"
server_key_iv: "f3a4e82740f67fa2bf197662277646d1"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.640362 16846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "6eeb517267474aafaba6681953067269"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "6f64a829bad175ccdc9ac07e0ae8c69c"
server_key_iv: "f3a4e82740f67fa2bf197662277646d1"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.643323 16846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:57:36.645473 16861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.646437 16846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:36.646554 16846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "6eeb517267474aafaba6681953067269"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "6f64a829bad175ccdc9ac07e0ae8c69c"
server_key_iv: "f3a4e82740f67fa2bf197662277646d1"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.646673 16846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:36.677944 16846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:36.680397 16846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:36.680575 16846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:36.681056 16846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:36.681752 16846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:36.681838 16846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.681913 16846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:36.681960 16846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.690003 16846 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:40103
I20260619 14:57:36.690033 16974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:40103 every 8 connection(s)
I20260619 14:57:36.690768 16846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:36.695150 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16846
I20260619 14:57:36.695240 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:36.695528 10526 external_mini_cluster.cc:1428] Setting key 454e820390fb5fe6f6b0ea5420c2ecb6
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:36.703101 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.692255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:35793 (local address 127.10.71.190:36521)
0619 14:57:36.692407 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:36.692410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:36.693062 (+ 652us) server_negotiation.cc:407] Connection header received
0619 14:57:36.694300 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:36.694303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:36.694357 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:36.694432 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:36.695480 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.696011 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.697035 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.697257 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.697360 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.699762 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:36.699765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:36.699767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:36.699787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:36.701149 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.701747 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.701749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.701750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.701813 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.702177 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.702179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.702180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.702330 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:36.702396 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:36.702920 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 14:57:36.702999 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:57:36.704118 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.692556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:35793)
0619 14:57:36.692915 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:36.692948 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:36.694101 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:36.694578 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:36.694595 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:36.694883 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:36.695345 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:36.695365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.696126 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.696129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:36.696913 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:36.696922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.697389 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.697393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:36.697550 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.698010 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:36.698023 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:36.699609 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:36.701284 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:36.701291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:36.701303 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:36.701656 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:36.701898 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:36.701906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:36.701910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:36.702076 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:36.702427 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:36.702434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:36.702699 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 14:57:36.702890 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 14:57:36.703172 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:57:36.705170 16975 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36521
I20260619 14:57:36.705364 16975 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:36.705807 16975 heartbeater.cc:511] Master 127.10.71.190:36521 requested a full tablet report, sending...
I20260619 14:57:36.706781 16650 ts_manager.cc:194] Registered new tserver with Master: 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.131@KRBTEST.COM" created.
I20260619 14:57:36.707356 16650 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:35793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab.
Entry for principal oryx/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab.
I20260619 14:57:36.718730 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36521
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:36.817899 16980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:36.818114 16980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:36.818171 16980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:36.821621 16980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:36.821687 16980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:36.821764 16980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:36.826458 16980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:36.827227 16980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:36.828058 16980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:36.836423 16985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.836474 16980 server_base.cc:1034] running on GCE node
W20260619 14:57:36.836450 16986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:36.836424 16988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:36.837092 16980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:36.837731 16980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:36.838907 16980 hybrid_clock.cc:648] HybridClock initialized: now 1781881056838886 us; error 41 us; skew 500 ppm
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:36.841320 16980 init.cc:377] Logged in from keytab as oryx/127.10.71.131@KRBTEST.COM (short username oryx)
I20260619 14:57:36.842041 16980 webserver.cc:460] Webserver started at http://127.10.71.131:41327/ using document root <none> and password file <none>
I20260619 14:57:36.842530 16980 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:36.842619 16980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:36.842820 16980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:36.844704 16980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "8e0b534270fb49cd8d5016202e853065"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "d291b9e0199b21575a706b13fb5c4c01"
server_key_iv: "5003687ed4c60da8ea7ef930a134eab1"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.845211 16980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "8e0b534270fb49cd8d5016202e853065"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "d291b9e0199b21575a706b13fb5c4c01"
server_key_iv: "5003687ed4c60da8ea7ef930a134eab1"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.848632 16980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:36.850986 16995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.851995 16980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:36.852133 16980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "8e0b534270fb49cd8d5016202e853065"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "d291b9e0199b21575a706b13fb5c4c01"
server_key_iv: "5003687ed4c60da8ea7ef930a134eab1"
server_key_version: "encryptionkey@0"
I20260619 14:57:36.852237 16980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:36.866945 16980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:36.869431 16980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:36.869635 16980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:36.870280 16980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:36.871024 16980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:36.871091 16980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.871150 16980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:36.871181 16980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:36.879431 16980 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:42443
I20260619 14:57:36.879467 17108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:42443 every 8 connection(s)
I20260619 14:57:36.880163 16980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:36.883255 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 16980
I20260619 14:57:36.883369 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:36.883653 10526 external_mini_cluster.cc:1428] Setting key f8bb93ca33b10b7d705a4139d176662b
Jun 19 14:57:36 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:36.892802 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.881731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:41669 (local address 127.10.71.190:36521)
0619 14:57:36.881849 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:36.881852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:36.882503 (+ 651us) server_negotiation.cc:407] Connection header received
0619 14:57:36.884063 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:36.884066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:36.884125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:36.884234 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:36.885369 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.885913 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.887006 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.887233 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.887341 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.889555 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:36.889559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:36.889563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:36.889589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:36.890944 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.891424 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.891429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.891431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.891507 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:36.891822 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:36.891827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:36.891829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:36.892001 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:36.892097 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:36.892553 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 14:57:36.892656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:36.893496 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:36.881966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:41669)
0619 14:57:36.882371 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:36.882406 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:36.883844 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:36.884407 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:36.884413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:36.884846 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:36.885239 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:36.885251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.886039 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.886041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:36.886880 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:36.886889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:36.887371 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:36.887375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:36.887525 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:36.888038 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:36.888052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:36.889404 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:36.891070 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:36.891075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:36.891087 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:36.891322 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:36.891595 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:36.891599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:36.891601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:36.891729 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:36.892097 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:36.892102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:36.892331 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 14:57:36.892515 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 14:57:36.892772 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260619 14:57:36.894438 17109 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36521
I20260619 14:57:36.894690 17109 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:36.895058 17109 heartbeater.cc:511] Master 127.10.71.190:36521 requested a full tablet report, sending...
I20260619 14:57:36.895896 16650 ts_manager.cc:194] Registered new tserver with Master: 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443)
I20260619 14:57:36.896730 16650 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:41669
I20260619 14:57:36.897245 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.71.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.189.keytab.
Entry for principal oryx/127.10.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.189.keytab.
I20260619 14:57:36.920608 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:38191
--webserver_interface=127.10.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.189.keytab
--principal=oryx/127.10.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:36521,127.10.71.189:38191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:37.016603 17114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:37.016781 17114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:37.016812 17114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:37.020452 17114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:37.020525 17114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:37.020556 17114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:37.020605 17114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:37.020619 17114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:37.025652 17114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:36521,127.10.71.189:38191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.189.keytab
--principal=oryx/127.10.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:38191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
W20260619 14:57:37.025782 17114 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.71.190:36521,127.10.71.189:38191'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:57:37.026463 17114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:37.027266 17114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:37.035045 17122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.035086 17119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.035086 17120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.035130 17114 server_base.cc:1034] running on GCE node
I20260619 14:57:37.035812 17114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:37.036799 17114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:37.037992 17114 hybrid_clock.cc:648] HybridClock initialized: now 1781881057037963 us; error 40 us; skew 500 ppm
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:37.040755 17114 init.cc:377] Logged in from keytab as oryx/127.10.71.189@KRBTEST.COM (short username oryx)
I20260619 14:57:37.041456 17114 webserver.cc:460] Webserver started at http://127.10.71.189:44285/ using document root <none> and password file <none>
I20260619 14:57:37.041887 17114 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.041940 17114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.042086 17114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:37.043854 17114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/instance:
uuid: "16e745a031fe4a348f1063ffd5df6651"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "c621670695af61880df5c4172cb6e054"
server_key_iv: "76d860d524f8b0ec803dbfaf4904c2e1"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.044428 17114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal/instance:
uuid: "16e745a031fe4a348f1063ffd5df6651"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "c621670695af61880df5c4172cb6e054"
server_key_iv: "76d860d524f8b0ec803dbfaf4904c2e1"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.047719 17114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:57:37.049996 17129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.051096 17114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:37.051216 17114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
uuid: "16e745a031fe4a348f1063ffd5df6651"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "c621670695af61880df5c4172cb6e054"
server_key_iv: "76d860d524f8b0ec803dbfaf4904c2e1"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.051307 17114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.064968 17114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:37.068010 17114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:37.068198 17114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:37.075336 17114 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.189:38191
I20260619 14:57:37.075596 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.189:38191 every 8 connection(s)
I20260619 14:57:37.076325 17114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
I20260619 14:57:37.078729 17182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:37.081797 17182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:57:37.085239 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 17114
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.189@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
I20260619 14:57:37.094651 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.083263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.10.71.190:36521)
0619 14:57:37.083424 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.083430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.084157 (+ 727us) server_negotiation.cc:407] Connection header received
0619 14:57:37.084956 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.084959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.085005 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.085103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.086455 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.087162 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.088419 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.088756 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.088838 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.091158 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.091160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.091162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.091183 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.092618 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.093158 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.093165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.093166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.093236 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.093604 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.093606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.093607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.093794 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.093865 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.094358 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.094472 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:57:37.095366 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.083610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52124)
0619 14:57:37.084006 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.084041 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.084774 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.085293 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.085301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.085737 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.086257 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.086273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.087307 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.087311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.088243 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.088252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.088880 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.088883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.089059 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.089580 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.089603 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.091009 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.092711 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.092722 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.092731 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.093035 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.093338 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.093341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.093343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.093513 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.093892 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.093898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.094138 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.094307 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.094588 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260619 14:57:37.095477 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.086076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.10.71.189:38191)
0619 14:57:37.086480 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.086493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.086518 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:57:37.086562 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.086566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.086738 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.086883 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.087609 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.088563 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.089592 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.089891 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.089966 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.092153 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.092157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.092164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.092181 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.093904 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.094277 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.094284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.094289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.094366 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.094675 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.094678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.094679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.094891 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.095052 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.095205 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.095309 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":305,"thread_start_us":161,"threads_started":1}
I20260619 14:57:37.096443 17182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.189@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
I20260619 14:57:37.103195 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.096812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44346)
0619 14:57:37.096910 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.096923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.096990 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.097235 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.097238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.097397 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.097471 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.097476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.098064 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.098067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.098503 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.098509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.099273 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.099275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.099338 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.099661 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.099671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.100639 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.102178 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.102180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.102181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.102362 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.102566 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.102567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.102568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.102630 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.102900 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.102902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.102962 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.103028 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.103107 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:57:37.103266 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.096948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.10.71.189:38191)
0619 14:57:37.097050 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.097054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.097078 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:57:37.097103 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.097105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.097132 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.097189 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.097556 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.097954 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.098587 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.099190 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.099258 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.100736 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.100738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.100740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.100756 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.102071 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.102431 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.102433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.102434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.102491 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.102693 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.102695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.102696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.102820 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.102882 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.103086 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.103156 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:57:37.105465 17182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Bootstrap starting.
I20260619 14:57:37.107204 17182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:37.107808 17182 log.cc:824] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:37.109344 17182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: No bootstrap required, opened a new log
I20260619 14:57:37.111840 17182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } }
I20260619 14:57:37.111999 17182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:37.112054 17182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e745a031fe4a348f1063ffd5df6651, State: Initialized, Role: FOLLOWER
I20260619 14:57:37.112504 17182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } }
I20260619 14:57:37.112883 17188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } } }
I20260619 14:57:37.113024 17188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:37.113498 17182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:57:37.117602 17199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:57:37.117677 17199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:57:37.118765 17114 master_runner.cc:428] Detected that this master 16e745a031fe4a348f1063ffd5df6651 is joining an existing cluster
I20260619 14:57:37.118826 17114 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.71.189:38191
I20260619 14:57:37.127871 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.120755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.10.71.190:36521)
0619 14:57:37.120840 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.120843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.121000 (+ 157us) server_negotiation.cc:407] Connection header received
0619 14:57:37.121121 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.121122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.121150 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.121232 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.121685 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.122329 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.123209 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.123441 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.123567 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.124762 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.124765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.124767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.124787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.126263 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.126751 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.126754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.126756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.126831 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.127135 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.127139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.127141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.127288 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.127383 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.127636 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.127726 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 14:57:37.127892 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.120453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52126)
0619 14:57:37.120940 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.120952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.121038 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.121246 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.121249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.121432 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.121567 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.121573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.122436 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.122439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.123106 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.123113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.123582 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.123585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.123705 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.124142 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.124160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.124613 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.126376 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.126379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.126380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.126644 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.126931 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.126935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.126940 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.127029 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.127419 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.127422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.127499 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.127587 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.127695 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260619 14:57:37.128604 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.120527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.10.71.189:38191)
0619 14:57:37.120613 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.120616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.121293 (+ 677us) server_negotiation.cc:407] Connection header received
0619 14:57:37.121419 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.121423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.121459 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.121532 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.122006 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.122656 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.123716 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.124083 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.124188 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.125256 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.125261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.125264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.125286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.126897 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.127372 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.127375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.127377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.127446 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.127726 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.127729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.127730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.127930 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.128061 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.128337 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.128454 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":20}
I20260619 14:57:37.128978 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.120535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44360)
0619 14:57:37.121233 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.121244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.121334 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.121579 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.121582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.121775 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.121893 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.121900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.122755 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.122758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.123606 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.123616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.124205 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.124209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.124336 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.124800 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.124813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.125144 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.127003 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.127025 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.127028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.127276 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.127531 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.127533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.127535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.127636 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.128114 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.128116 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.128186 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.128741 (+ 555us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.128841 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":104,"threads_started":1}
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.190@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
I20260619 14:57:37.139357 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.131644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44372)
0619 14:57:37.131904 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.131935 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.132014 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.132402 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.132406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.132657 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.132732 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.132737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.133620 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.133622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.134183 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.134190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.134735 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.134737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.134800 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.135199 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.135209 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.136518 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.138086 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.138093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.138096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.138276 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.138519 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.138521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.138522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.138590 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.138923 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.138927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.139001 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.139141 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.139254 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260619 14:57:37.139428 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.131675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.10.71.189:38191)
0619 14:57:37.132142 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.132146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.132157 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:37.132205 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.132208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.132244 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.132337 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.132840 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.133512 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.134286 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.134631 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.134730 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.136638 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.136642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.136644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.136665 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.137981 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.138362 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.138366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.138368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.138439 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.138664 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.138667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.138668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.138837 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.138921 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.139211 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.139304 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260619 14:57:37.140686 16649 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.71.189:38191: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "16e745a031fe4a348f1063ffd5df6651"
member_type: NON_VOTER
last_known_addr {
host: "127.10.71.189"
port: 38191
}
attrs {
promote: true
}
}
dest_uuid: "359478d2a1624cadbfdc4d89d8997014"
cas_config_opid_index: -1
I20260619 14:57:37.141224 16649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } }
I20260619 14:57:37.142438 17209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.143033 17156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:37.143747 17209 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.71.189:38191
I20260619 14:57:37.143935 17209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.144028 17209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:37.144176 17156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Refusing update from remote peer 359478d2a1624cadbfdc4d89d8997014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:57:37.144789 17208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:37.144878 17114 master.cc:537] Master@127.10.71.189:38191 shutting down...
W20260619 14:57:37.146250 16637 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 14:57:37.147032 16637 consensus_peers.cc:487] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 -> Peer 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189:38191): Couldn't send request to peer 16e745a031fe4a348f1063ffd5df6651. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:57:37.148778 17114 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:37.149019 17114 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:37.149250 17114 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: stopping tablet replica
W20260619 14:57:37.152281 16603 connection.cc:535] client connection to 127.10.71.189:38191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:57:37.163710 17114 master.cc:552] Master@127.10.71.189:38191 shutdown complete.
I20260619 14:57:37.163779 17114 master_runner.cc:305] Clearing existing system tablet
I20260619 14:57:37.163986 17114 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.164069 17114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.165241 17114 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:37.166045 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.166244 17114 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.166363 17114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:37.166424 17114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
uuid: "16e745a031fe4a348f1063ffd5df6651"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "c621670695af61880df5c4172cb6e054"
server_key_iv: "76d860d524f8b0ec803dbfaf4904c2e1"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.167857 17114 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:37.169986 17114 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:57:37.170100 17114 log.cc:1196] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:57:37.170362 17114 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Deleting consensus metadata
I20260619 14:57:37.170539 17114 master_runner.cc:315] Copying system tablet from 127.10.71.190:36521
I20260619 14:57:37.171633 17114 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: tablet copy: Beginning tablet copy session from remote peer at address 127.10.71.190:36521
I20260619 14:57:37.179014 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.172149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.10.71.190:36521)
0619 14:57:37.172266 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.172270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.172505 (+ 235us) server_negotiation.cc:407] Connection header received
0619 14:57:37.172630 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.172633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.172671 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.172768 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.173199 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.173738 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.174575 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.174815 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.174923 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.176045 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.176049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.176051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.176075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.177450 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.177876 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.177881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.177885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.177965 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.178250 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.178253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.178254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.178411 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.178516 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.178774 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.178869 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:37.179024 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.172103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52136)
0619 14:57:37.172420 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.172431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.172544 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.172794 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.172797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.172978 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.173085 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.173091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.173852 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.173859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.174444 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.174453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.174923 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.174927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.175021 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.175555 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.175566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.175916 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.177581 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.177586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.177588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.177778 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.178069 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.178072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.178074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.178155 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.178507 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.178510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.178601 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.178712 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.178840 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20260619 14:57:37.179531 16670 tablet_copy_service.cc:140] P 359478d2a1624cadbfdc4d89d8997014: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 16e745a031fe4a348f1063ffd5df6651 ({username='oryx', principal='oryx/127.10.71.189@KRBTEST.COM'} at 127.0.0.1:52136)
I20260619 14:57:37.179678 16670 tablet_copy_service.cc:161] P 359478d2a1624cadbfdc4d89d8997014: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 16e745a031fe4a348f1063ffd5df6651 at {username='oryx', principal='oryx/127.10.71.189@KRBTEST.COM'} at 127.0.0.1:52136: session id = 16e745a031fe4a348f1063ffd5df6651-00000000000000000000000000000000
I20260619 14:57:37.181169 16670 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:57:37.182345 17114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:57:37.183714 17114 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: tablet copy: Starting download of 0 data blocks...
I20260619 14:57:37.183892 17114 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: tablet copy: Starting download of 1 WAL segments...
I20260619 14:57:37.185299 17114 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:57:37.186676 16670 tablet_copy_service.cc:342] P 359478d2a1624cadbfdc4d89d8997014: Request end of tablet copy session 16e745a031fe4a348f1063ffd5df6651-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.71.189@KRBTEST.COM'} at 127.0.0.1:52136
I20260619 14:57:37.186774 16670 tablet_copy_service.cc:433] P 359478d2a1624cadbfdc4d89d8997014: ending tablet copy session 16e745a031fe4a348f1063ffd5df6651-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 16e745a031fe4a348f1063ffd5df6651
I20260619 14:57:37.192512 17114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:37.193943 17220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.194031 17223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.194041 17221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.194243 17114 server_base.cc:1034] running on GCE node
I20260619 14:57:37.194375 17114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:37.194648 17114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:37.195787 17114 hybrid_clock.cc:648] HybridClock initialized: now 1781881057195768 us; error 32 us; skew 500 ppm
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:37.196941 17114 init.cc:377] Logged in from keytab as oryx/127.10.71.189@KRBTEST.COM (short username oryx)
I20260619 14:57:37.197343 17114 webserver.cc:460] Webserver started at http://127.10.71.189:46147/ using document root <none> and password file <none>
I20260619 14:57:37.197499 17114 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.197546 17114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.198397 17114 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:37.199128 17230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.199321 17114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:37.199383 17114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
uuid: "16e745a031fe4a348f1063ffd5df6651"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "c621670695af61880df5c4172cb6e054"
server_key_iv: "76d860d524f8b0ec803dbfaf4904c2e1"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.199455 17114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.207922 17114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:37.209548 17114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:37.213588 17114 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.189:38191
I20260619 14:57:37.213770 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.189:38191 every 8 connection(s)
I20260619 14:57:37.214092 17114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
I20260619 14:57:37.215227 17283 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:57:37.215911 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Bootstrap starting.
I20260619 14:57:37.223161 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:57:37.223438 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Bootstrap complete.
I20260619 14:57:37.223870 17283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } }
I20260619 14:57:37.223966 17283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16e745a031fe4a348f1063ffd5df6651, State: Initialized, Role: LEARNER
I20260619 14:57:37.224074 17283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } }
I20260619 14:57:37.224399 17286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.224489 17283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:37.224509 17286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:57:37.226974 17297 catalog_manager.cc:1259] Loaded cluster ID: 262ce88f198349a4aae271a44ca707fc
I20260619 14:57:37.227048 17297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: loading cluster ID for follower catalog manager: success
I20260619 14:57:37.228535 17297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: acquiring CA information for follower catalog manager: success
I20260619 14:57:37.229483 17297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:57:37.262543 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.253695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.10.71.189:38191)
0619 14:57:37.253893 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.253898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.253915 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:57:37.254096 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.254099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.254144 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.254235 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.255137 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.256125 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.257175 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.257488 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.257997 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.259250 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.259253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.259255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.259290 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.261130 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.261484 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.261487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.261489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.261562 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.261786 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.261789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.261790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.261963 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.262089 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.262287 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.262394 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for oryx/127.10.71.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.188.keytab.
Entry for principal oryx/127.10.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.188.keytab.
I20260619 14:57:37.285151 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:42157
--webserver_interface=127.10.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.188.keytab
--principal=oryx/127.10.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:37.380419 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:37.380601 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:37.380633 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:37.384173 17301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:37.384223 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:37.384243 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:37.384258 17301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:37.384274 17301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:37.388815 17301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.188.keytab
--principal=oryx/127.10.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:37.389504 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:37.390272 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:37.396894 17309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.396908 17306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.396943 17307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.397032 17301 server_base.cc:1034] running on GCE node
I20260619 14:57:37.397491 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:37.398312 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:37.399479 17301 hybrid_clock.cc:648] HybridClock initialized: now 1781881057399459 us; error 24 us; skew 500 ppm
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:37.401896 17301 init.cc:377] Logged in from keytab as oryx/127.10.71.188@KRBTEST.COM (short username oryx)
I20260619 14:57:37.402526 17301 webserver.cc:460] Webserver started at http://127.10.71.188:35711/ using document root <none> and password file <none>
I20260619 14:57:37.402918 17301 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.402967 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.403105 17301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:37.404824 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/instance:
uuid: "de3417a8d9ef473daf305799cc15e3e2"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "ef2da86891a557cbe4a42402077fd9a4"
server_key_iv: "d4ebeec1831ffb86d3ccedff82dad3bf"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.405244 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal/instance:
uuid: "de3417a8d9ef473daf305799cc15e3e2"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "ef2da86891a557cbe4a42402077fd9a4"
server_key_iv: "d4ebeec1831ffb86d3ccedff82dad3bf"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.408174 17301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:37.410182 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.411149 17301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:37.411230 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
uuid: "de3417a8d9ef473daf305799cc15e3e2"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "ef2da86891a557cbe4a42402077fd9a4"
server_key_iv: "d4ebeec1831ffb86d3ccedff82dad3bf"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.411294 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.419349 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:37.421820 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:37.421938 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:37.428694 17301 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.188:42157
I20260619 14:57:37.428970 17368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.188:42157 every 8 connection(s)
I20260619 14:57:37.429554 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
I20260619 14:57:37.432065 17369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:37.435441 17369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:57:37.439599 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 17301
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.188@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.71.188@KRBTEST.COM
I20260619 14:57:37.448223 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.437136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.10.71.190:36521)
0619 14:57:37.437274 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.437278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.437865 (+ 587us) server_negotiation.cc:407] Connection header received
0619 14:57:37.438640 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.438643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.438685 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.438743 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.439829 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.440541 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.441582 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.441812 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.441925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.444423 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.444426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.444428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.444452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.445970 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.446464 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.446468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.446469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.446566 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.447123 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.447126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.447127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.447333 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.447439 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.447996 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.448110 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:57:37.449184 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.437331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52148)
0619 14:57:37.437721 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.437759 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.438455 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.438878 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.438883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.439166 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.439648 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.439660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.440684 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.440686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.441460 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.441468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.441964 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.441968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.442168 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.442666 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.442687 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.444277 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.446092 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.446100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.446115 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.446349 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.446653 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.446656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.446658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.446826 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.447459 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.447464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.447727 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.447942 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.448262 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260619 14:57:37.450621 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.440382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.10.71.188:42157)
0619 14:57:37.440702 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.440717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.440741 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:57:37.440785 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.440799 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.440939 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.441091 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.441756 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.442773 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.443717 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.444105 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.444194 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.446602 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.446607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.446634 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.446656 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.448348 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.448816 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.448820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.448822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.448908 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.449586 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.449589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.449590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.449896 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.450061 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.450300 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.450419 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":140,"threads_started":1}
I20260619 14:57:37.450723 17369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.188@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
I20260619 14:57:37.457818 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.451089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44400)
0619 14:57:37.451190 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.451203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.451400 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.451655 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.451657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.451790 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.451876 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.451881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.452664 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.452667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.453108 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.453114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.453497 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.453499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.453576 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.453990 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.454000 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.455134 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.456781 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.456784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.456785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.456951 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.457171 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.457172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.457174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.457238 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.457518 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.457520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.457582 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.457653 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.457733 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:57:37.457926 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.451211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.10.71.189:38191)
0619 14:57:37.451330 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.451335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.451349 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:37.451510 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.451513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.451552 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.451623 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.451979 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.452572 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.453199 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.453395 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.453485 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.455269 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.455272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.455274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.455295 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.456693 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.457029 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.457031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.457032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.457099 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.457308 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.457310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.457311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.457449 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.457516 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.457696 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.457778 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:57:37.458513 17369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.188@KRBTEST.COM for oryx/127.10.71.188@KRBTEST.COM
I20260619 14:57:37.464715 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.458855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.0.0.1:58456)
0619 14:57:37.458958 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.458969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.459025 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.459210 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.459212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.459313 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.459367 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.459371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.459935 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.459937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.460414 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.460421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.460859 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.460861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.460906 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.461281 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.461290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.462217 (+ 927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.463593 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.463596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.463597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.463764 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.463987 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.463989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.463990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.464053 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.464396 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.464398 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.464461 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.464538 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.464616 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 14:57:37.464794 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.458892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.10.71.188:42157)
0619 14:57:37.458994 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.458997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.459009 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:37.459099 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.459101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.459130 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.459186 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.459445 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.459856 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.460505 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.460776 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.460848 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.462311 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.462313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.462315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.462331 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.463503 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.463844 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.463846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.463847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.463911 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.464133 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.464135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.464136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.464269 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.464414 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.464594 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.464671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:57:37.467346 17369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Bootstrap starting.
I20260619 14:57:37.469559 17369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:37.470258 17369 log.cc:824] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:37.471958 17369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: No bootstrap required, opened a new log
I20260619 14:57:37.474820 17369 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } }
I20260619 14:57:37.474985 17369 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:37.475016 17369 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3417a8d9ef473daf305799cc15e3e2, State: Initialized, Role: FOLLOWER
I20260619 14:57:37.475333 17369 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } }
I20260619 14:57:37.475816 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } } }
I20260619 14:57:37.475943 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:37.476464 17369 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:57:37.480716 17385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:57:37.480799 17385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:57:37.484048 17301 master_runner.cc:428] Detected that this master de3417a8d9ef473daf305799cc15e3e2 is joining an existing cluster
I20260619 14:57:37.484105 17301 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.71.188:42157
I20260619 14:57:37.503239 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16712
I20260619 14:57:37.508073 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.486563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.10.71.190:36521)
0619 14:57:37.486673 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.486676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.486687 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:37.486725 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.486727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.486758 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.486849 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.487362 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.488208 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.489437 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.489766 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.489839 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.504808 (+ 14969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.504812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.504814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.504838 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.506275 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.506778 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.506782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.506783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.506864 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.507159 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.507162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.507164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.507353 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.507416 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.507768 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.507896 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:37.509153 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.486039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52154)
0619 14:57:37.486458 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.486473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.486557 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.486864 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.486867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.487074 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.487219 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.487228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.488343 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.488346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.489321 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.489329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.489996 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.490000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.490091 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.490546 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.490558 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.504628 (+ 14070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.506385 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.506388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.506390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.506663 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.506966 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.506969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.506971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.507064 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.507500 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.507503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.507595 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.508428 (+ 833us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.508582 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":55,"thread_start_us":115,"threads_started":1}
I20260619 14:57:37.510075 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.486581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.10.71.188:42157)
0619 14:57:37.486671 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.486675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.487453 (+ 778us) server_negotiation.cc:407] Connection header received
0619 14:57:37.487569 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.487571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.487600 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.487642 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.488177 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.488914 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.490038 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.490346 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.490397 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.491392 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.491395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.491397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.491417 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.506171 (+ 14754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.508637 (+ 2466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.508643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.508648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.508734 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.509315 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.509319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.509323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.509509 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.509587 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.509852 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.509954 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:57:37.510423 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.486742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.0.0.1:58462)
0619 14:57:37.487374 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.487386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.487494 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.487736 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.487739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.487971 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.488076 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.488082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.489049 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.489051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.489938 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.489946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.490542 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.490544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.490635 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.491008 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.491018 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.491280 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.506834 (+ 15554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.506837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.506840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.508530 (+ 1690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.508838 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.508843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.508848 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.508939 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.509645 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.509647 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.509715 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.510174 (+ 459us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.510284 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":42,"thread_start_us":63,"threads_started":1}
I20260619 14:57:37.510691 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.486193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.10.71.189:38191)
0619 14:57:37.486479 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.486485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.487159 (+ 674us) server_negotiation.cc:407] Connection header received
0619 14:57:37.487227 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.487230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.487276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.487370 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.487982 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.488793 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.490156 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.491032 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.491103 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.506459 (+ 15356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.506465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.506470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.506497 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.508257 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.508923 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.508927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.508928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.509022 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.509608 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.509611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.509613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.509814 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.509881 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.510426 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.510534 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260619 14:57:37.512148 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.486182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44406)
0619 14:57:37.487044 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.487058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.487151 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.487499 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.487502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.487712 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.487849 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.487856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.488928 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.488931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.490023 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.490036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.491238 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.491241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.491325 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.491681 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.491695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.506341 (+ 14646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.508400 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.508406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.508411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.508810 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.509121 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.509124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.509126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.509480 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.509950 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.509952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.510027 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.511871 (+ 1844us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.511991 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":51,"thread_start_us":83,"threads_started":1}
I20260619 14:57:37.513957 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:46755
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:36521
--webserver_port=44215
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881056, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.190@KRBTEST.COM for oryx/127.10.71.188@KRBTEST.COM
I20260619 14:57:37.520242 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.512568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.0.0.1:58464)
0619 14:57:37.512813 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.512824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.512922 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.513205 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.513209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.513375 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.513481 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.513487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.514370 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.514373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.515206 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.515215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.515750 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.515753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.515841 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.516243 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.516255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.517452 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.518946 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.518949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.518951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.519128 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.519376 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.519380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.519382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.519459 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.519798 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.519802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.519894 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.519999 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.520114 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260619 14:57:37.520267 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.512452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.10.71.188:42157)
0619 14:57:37.512746 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.512749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.512886 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:57:37.513037 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.513042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.513109 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.513185 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.513601 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.514272 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.515317 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.515639 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.515741 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.517591 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.517595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.517597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.517617 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.518832 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.519223 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.519226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.519228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.519294 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.519547 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.519549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.519551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.519722 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.519795 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.520085 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.520166 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260619 14:57:37.521165 16649 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.71.188:42157: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2"
member_type: NON_VOTER
last_known_addr {
host: "127.10.71.188"
port: 42157
}
attrs {
promote: true
}
}
dest_uuid: "359478d2a1624cadbfdc4d89d8997014"
cas_config_opid_index: 5
I20260619 14:57:37.521629 16649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } }
I20260619 14:57:37.522490 17208 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.523527 17208 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.71.188:42157
I20260619 14:57:37.523697 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.523809 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:37.524638 17343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:37.525763 17343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Refusing update from remote peer 359478d2a1624cadbfdc4d89d8997014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:57:37.526487 17209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:37.527519 17301 master.cc:537] Master@127.10.71.188:42157 shutting down...
I20260619 14:57:37.530186 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.526102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.10.71.189:38191)
0619 14:57:37.526220 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.526224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.526236 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:37.526267 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.526269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.526317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.526391 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:37.526641 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.527251 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.528862 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.529427 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.529505 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.529626 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.530010 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.530086 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:57:37.530628 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.524149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44422)
0619 14:57:37.524240 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.524253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.524341 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.526420 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.526423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.526437 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:37.526506 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.526512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.527367 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.527370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.528591 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.528600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.529796 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.529799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.529879 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.529912 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.530355 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.530431 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:57:37.530934 17343 raft_consensus.cc:858] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } }
W20260619 14:57:37.531021 17343 raft_consensus.cc:1565] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } }
I20260619 14:57:37.531101 17257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 LEARNER]: Refusing update from remote peer 359478d2a1624cadbfdc4d89d8997014: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:57:37.532078 17209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 14:57:37.532277 17343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.533705 17257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.535671 17286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359478d2a1624cadbfdc4d89d8997014. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.535775 17286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:57:37.535866 17286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.535933 17286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:57:37.536033 17395 mvcc.cc:205] Tried to move back new op lower bound from 7298584806649712640 to 7298584806476886016. Current Snapshot: MvccSnapshot[applied={T|T < 7298584806505558016}]
I20260619 14:57:37.536204 17399 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.536302 17399 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 14:57:37.536392 17399 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.536448 17399 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 14:57:37.536434 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359478d2a1624cadbfdc4d89d8997014. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: NON_VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: true } } }
I20260619 14:57:37.536506 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 14:57:37.537760 17209 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: attempting to promote NON_VOTER 16e745a031fe4a348f1063ffd5df6651 to VOTER
I20260619 14:57:37.538079 17209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } }
I20260619 14:57:37.539186 17257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 LEARNER]: Refusing update from remote peer 359478d2a1624cadbfdc4d89d8997014: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:57:37.539605 17208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 14:57:37.539951 16637 consensus_peers.cc:487] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 -> Peer de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188:42157): Couldn't send request to peer de3417a8d9ef473daf305799cc15e3e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:57:37.541285 17208 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.542343 17301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 14:57:37.542568 17301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 14:57:37.542591 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.542645 17301 tablet_replica.cc:330] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: stopping tablet replica
I20260619 14:57:37.542673 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:37.542778 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.542835 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:37.543655 17257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.545928 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.546288 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:37.557621 17301 master.cc:552] Master@127.10.71.188:42157 shutdown complete.
I20260619 14:57:37.557679 17301 master_runner.cc:305] Clearing existing system tablet
I20260619 14:57:37.557869 17301 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.557928 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.558776 17301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:37.559499 17406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.559669 17301 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.559772 17301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:37.559809 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
uuid: "de3417a8d9ef473daf305799cc15e3e2"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "ef2da86891a557cbe4a42402077fd9a4"
server_key_iv: "d4ebeec1831ffb86d3ccedff82dad3bf"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.560971 17301 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:37.562414 17301 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:57:37.562500 17301 log.cc:1196] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:57:37.562729 17301 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Deleting consensus metadata
I20260619 14:57:37.562873 17301 master_runner.cc:315] Copying system tablet from 127.10.71.190:36521
I20260619 14:57:37.563810 17301 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: tablet copy: Beginning tablet copy session from remote peer at address 127.10.71.190:36521
I20260619 14:57:37.571049 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.564341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.10.71.190:36521)
0619 14:57:37.564457 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.564463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.564628 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:57:37.564753 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.564756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.564791 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.564867 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.565226 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.565670 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.566416 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.566635 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.566732 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.567705 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.567708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.567711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.567732 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.569433 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.569865 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.569868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.569870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.569945 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.570253 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.570257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.570259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.570417 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.570513 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.570843 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.570935 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:57:37.571058 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.564272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52162)
0619 14:57:37.564538 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.564558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.564678 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.564879 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.564881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.565049 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.565130 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.565134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.565782 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.565787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.566307 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.566314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.566734 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.566737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.566828 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.567288 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.567301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.567582 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.569538 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.569542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.569544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.569770 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.570092 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.570094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.570098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.570168 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.570505 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.570551 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.570658 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.570769 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.570892 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260619 14:57:37.571672 16670 tablet_copy_service.cc:140] P 359478d2a1624cadbfdc4d89d8997014: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer de3417a8d9ef473daf305799cc15e3e2 ({username='oryx', principal='oryx/127.10.71.188@KRBTEST.COM'} at 127.0.0.1:52162)
I20260619 14:57:37.571774 16670 tablet_copy_service.cc:161] P 359478d2a1624cadbfdc4d89d8997014: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer de3417a8d9ef473daf305799cc15e3e2 at {username='oryx', principal='oryx/127.10.71.188@KRBTEST.COM'} at 127.0.0.1:52162: session id = de3417a8d9ef473daf305799cc15e3e2-00000000000000000000000000000000
I20260619 14:57:37.572646 16670 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:57:37.573619 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:57:37.575178 17301 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: tablet copy: Starting download of 0 data blocks...
I20260619 14:57:37.575366 17301 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: tablet copy: Starting download of 1 WAL segments...
I20260619 14:57:37.576581 17301 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:57:37.578155 16670 tablet_copy_service.cc:342] P 359478d2a1624cadbfdc4d89d8997014: Request end of tablet copy session de3417a8d9ef473daf305799cc15e3e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.71.188@KRBTEST.COM'} at 127.0.0.1:52162
I20260619 14:57:37.578241 16670 tablet_copy_service.cc:433] P 359478d2a1624cadbfdc4d89d8997014: ending tablet copy session de3417a8d9ef473daf305799cc15e3e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer de3417a8d9ef473daf305799cc15e3e2
I20260619 14:57:37.584254 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:37.585815 17416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.585969 17301 server_base.cc:1034] running on GCE node
W20260619 14:57:37.586104 17414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.586270 17413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.586450 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:37.586750 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:37.587913 17301 hybrid_clock.cc:648] HybridClock initialized: now 1781881057587924 us; error 40 us; skew 500 ppm
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:37.589295 17301 init.cc:377] Logged in from keytab as oryx/127.10.71.188@KRBTEST.COM (short username oryx)
I20260619 14:57:37.589708 17301 webserver.cc:460] Webserver started at http://127.10.71.188:38043/ using document root <none> and password file <none>
I20260619 14:57:37.589865 17301 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.589915 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.590860 17301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:37.591621 17423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.591831 17301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:37.591913 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
uuid: "de3417a8d9ef473daf305799cc15e3e2"
format_stamp: "Formatted at 2026-06-19 14:57:37 on dist-test-slave-3qdh"
server_key: "ef2da86891a557cbe4a42402077fd9a4"
server_key_iv: "d4ebeec1831ffb86d3ccedff82dad3bf"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.591998 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.597525 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:37.599314 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:37.603753 17301 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.188:42157
I20260619 14:57:37.604341 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
I20260619 14:57:37.605569 17476 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:57:37.606331 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Bootstrap starting.
I20260619 14:57:37.612092 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:57:37.612469 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: Bootstrap complete.
I20260619 14:57:37.612581 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.188:42157 every 8 connection(s)
I20260619 14:57:37.612970 17476 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } }
I20260619 14:57:37.613096 17476 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de3417a8d9ef473daf305799cc15e3e2, State: Initialized, Role: LEARNER
I20260619 14:57:37.613306 17476 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } }
I20260619 14:57:37.613658 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:37.613793 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:57:37.613860 17476 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:37.617022 17490 catalog_manager.cc:1259] Loaded cluster ID: 262ce88f198349a4aae271a44ca707fc
I20260619 14:57:37.617096 17490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: loading cluster ID for follower catalog manager: success
I20260619 14:57:37.618364 17490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: acquiring CA information for follower catalog manager: success
I20260619 14:57:37.619343 17490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:57:37.661067 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:37.661352 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:37.661407 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:37.666837 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:37.666904 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:37.666993 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:37.673601 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:46755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=44215
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:37.674415 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:37.675251 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:37.683806 17495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.683876 17394 server_base.cc:1034] running on GCE node
W20260619 14:57:37.683830 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.683810 17498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.684484 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:37.685066 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:37.686240 17394 hybrid_clock.cc:648] HybridClock initialized: now 1781881057686219 us; error 39 us; skew 500 ppm
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:37.688766 17394 init.cc:377] Logged in from keytab as oryx/127.10.71.129@KRBTEST.COM (short username oryx)
I20260619 14:57:37.689563 17394 webserver.cc:460] Webserver started at http://127.10.71.129:44215/ using document root <none> and password file <none>
I20260619 14:57:37.690114 17394 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.690205 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.693756 17394 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:37.695530 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.696576 17394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:37.696671 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "65529b9919b04d4b923c9b3c52f0855d"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "312abf441d1b9070cc640ec77afb8cf8"
server_key_iv: "ec874e74052735cd33721780e00a1cc0"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.697078 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.709407 16975 heartbeater.cc:503] Master 127.10.71.190:36521 was elected leader, sending a full tablet report...
I20260619 14:57:37.724097 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:37.727401 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:37.727584 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:37.728180 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:37.729012 17394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:37.729056 17394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.729139 17394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:37.729171 17394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.737177 17394 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:46755
I20260619 14:57:37.737248 17618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:46755 every 8 connection(s)
I20260619 14:57:37.738139 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:37.741470 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 17394
I20260619 14:57:37.741678 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16846
I20260619 14:57:37.749464 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:40103
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:36521
--webserver_port=46151
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.188@KRBTEST.COM
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
I20260619 14:57:37.755076 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.739817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:40407 (local address 127.10.71.190:36521)
0619 14:57:37.739957 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.739962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.741268 (+ 1306us) server_negotiation.cc:407] Connection header received
0619 14:57:37.742244 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.742250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.742314 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.742395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.744283 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.745252 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747139 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.747550 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.750739 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.750743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.750746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.750791 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.752610 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.753280 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.753284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.753287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.753372 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.753806 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.753813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.753817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.754048 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.754143 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.754780 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.754898 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:57:37.755676 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.739885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:37275 (local address 127.10.71.188:42157)
0619 14:57:37.740156 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.740160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.741032 (+ 872us) server_negotiation.cc:407] Connection header received
0619 14:57:37.742488 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.742494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.742558 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.742652 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.744436 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.745472 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747132 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.747547 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747658 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.751255 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.751262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.751265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.751292 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.753771 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.754204 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.754207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.754209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.754291 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.754609 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.754612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.754614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.754821 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.754905 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.755446 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.755560 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260619 14:57:37.756191 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.741697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:54801 (local address 127.10.71.189:38191)
0619 14:57:37.741815 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.741819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.741833 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:37.742974 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.742979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.743032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.743113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.744342 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.745179 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747701 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.748039 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.748126 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.752445 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.752451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.752456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.752486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.754279 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.754829 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.754836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.754839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.754927 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.755262 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.755265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.755267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.755514 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.755592 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.755942 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.756045 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:57:37.757020 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.740255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.10.71.129:37275)
0619 14:57:37.740841 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.740902 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.742379 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.742792 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.742797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.743275 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.744319 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.744325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.745589 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.745592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.746989 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.746999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.747827 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.748062 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.748887 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.748897 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.750866 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.753876 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.753879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.753881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.754095 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.754386 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.754389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.754403 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.754493 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.754947 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.754950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.755029 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.755150 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.756068 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":619,"spinlock_wait_cycles":14720,"thread_start_us":659,"threads_started":1}
I20260619 14:57:37.757287 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.741295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.10.71.129:54801)
0619 14:57:37.741558 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.741572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.742307 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.743258 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.743260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.743414 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.744184 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.744196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.745335 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.745338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.747567 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.747578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.748285 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.748288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.748432 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.750808 (+ 2376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.750826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.752277 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.754413 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.754417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.754422 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.754703 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.755053 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.755058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.755061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.755152 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.755686 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.755690 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.755787 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.755888 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.756163 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1580,"thread_start_us":81,"threads_started":1}
I20260619 14:57:37.757967 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.740275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:40407)
0619 14:57:37.741162 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.741172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.742037 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.742706 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.742717 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.743279 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.744117 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.744139 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.745398 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.745404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.746988 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.747001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.747844 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.747851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.748058 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.748736 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.748762 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.750512 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.752773 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.752779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.752794 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.753155 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.753506 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.753513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.753515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.753687 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.754222 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.754228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.754516 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.755430 (+ 914us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.757812 (+ 2382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":2168,"spinlock_wait_cycles":12800,"thread_start_us":125,"threads_started":1}
I20260619 14:57:37.758370 17621 heartbeater.cc:347] Connected to a master server at 127.10.71.189:38191
I20260619 14:57:37.758450 17619 heartbeater.cc:347] Connected to a master server at 127.10.71.188:42157
I20260619 14:57:37.758586 17621 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:37.758585 17619 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:37.758666 17620 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36521
I20260619 14:57:37.758718 17620 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:37.759166 17620 heartbeater.cc:511] Master 127.10.71.190:36521 requested a full tablet report, sending...
I20260619 14:57:37.759166 17621 heartbeater.cc:511] Master 127.10.71.189:38191 requested a full tablet report, sending...
I20260619 14:57:37.759184 17619 heartbeater.cc:511] Master 127.10.71.188:42157 requested a full tablet report, sending...
I20260619 14:57:37.760339 16649 ts_manager.cc:194] Re-registered known tserver with Master: 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755)
I20260619 14:57:37.760936 17440 ts_manager.cc:194] Registered new tserver with Master: 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755)
I20260619 14:57:37.761010 16649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:40407
I20260619 14:57:37.762188 17247 ts_manager.cc:194] Registered new tserver with Master: 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755)
W20260619 14:57:37.853760 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:37.853960 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:37.854034 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:37.857689 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:37.857730 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:37.857825 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:37.862083 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=46151
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:37.862879 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:37.863708 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:37.871968 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.872107 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:37.871968 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:37.872131 17627 server_base.cc:1034] running on GCE node
I20260619 14:57:37.872561 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:37.873087 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:37.874233 17627 hybrid_clock.cc:648] HybridClock initialized: now 1781881057874218 us; error 22 us; skew 500 ppm
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:37.876644 17627 init.cc:377] Logged in from keytab as oryx/127.10.71.130@KRBTEST.COM (short username oryx)
I20260619 14:57:37.877303 17627 webserver.cc:460] Webserver started at http://127.10.71.130:46151/ using document root <none> and password file <none>
I20260619 14:57:37.877754 17627 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:37.877817 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:37.881461 17627 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:57:37.883381 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.884514 17627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 14:57:37.884604 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "6eeb517267474aafaba6681953067269"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "6f64a829bad175ccdc9ac07e0ae8c69c"
server_key_iv: "f3a4e82740f67fa2bf197662277646d1"
server_key_version: "encryptionkey@0"
I20260619 14:57:37.884997 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:37.898599 17109 heartbeater.cc:503] Master 127.10.71.190:36521 was elected leader, sending a full tablet report...
I20260619 14:57:37.913827 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:37.916468 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:37.916666 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:37.917130 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:37.917793 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:37.917830 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.917886 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:37.917912 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:37.926350 17627 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:40103
I20260619 14:57:37.926410 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:40103 every 8 connection(s)
I20260619 14:57:37.927176 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:37.935858 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 17627
I20260619 14:57:37.936026 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16980
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.188@KRBTEST.COM
Jun 19 14:57:37 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881057, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
I20260619 14:57:37.940634 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.928685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56933 (local address 127.10.71.189:38191)
0619 14:57:37.928800 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.928804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.929670 (+ 866us) server_negotiation.cc:407] Connection header received
0619 14:57:37.930536 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.930539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.930593 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.930657 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.931691 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.932497 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.934180 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.934572 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.934675 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.937244 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.937247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.937250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.937279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.938743 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.939170 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.939173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.939175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.939269 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.939542 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.939545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.939546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.939757 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.939847 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.940398 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.940496 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:57:37.941365 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.929031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:56465)
0619 14:57:37.929809 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.929818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.930378 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.930800 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.930802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.931222 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.931561 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.931568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.932371 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.932374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.933590 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.933599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.934134 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.934140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.934251 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.934699 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.934713 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.936354 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.938324 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.938331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.938345 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.938712 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.938998 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.939003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.939008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.939184 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.940197 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.940199 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.940262 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.940374 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.940617 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":159,"spinlock_wait_cycles":4352,"thread_start_us":91,"threads_started":1}
I20260619 14:57:37.941689 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.929031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.10.71.130:56933)
0619 14:57:37.929486 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.929537 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.930363 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.930785 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.930790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.931162 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.931549 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.931559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.932598 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.932602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.933589 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.933599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.935374 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.935389 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.935457 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.935820 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.935833 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.937104 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.938869 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.938874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.938875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.939086 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.939335 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.939340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.939345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.939444 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.939898 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.939906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.940151 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.940883 (+ 732us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.940986 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":32,"thread_start_us":593,"threads_started":1}
I20260619 14:57:37.941926 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.928632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56465 (local address 127.10.71.190:36521)
0619 14:57:37.928763 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.928768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.930121 (+ 1353us) server_negotiation.cc:407] Connection header received
0619 14:57:37.930598 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.930601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.930648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.930704 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.931701 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.932217 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.933708 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.934012 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.934095 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.936548 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.936553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.936559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.936587 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.938219 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.938812 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.938817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.938822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.938904 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.939627 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.939630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.939632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.939814 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.939890 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.941690 (+ 1800us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.941794 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:57:37.942519 17757 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36521
I20260619 14:57:37.942708 17757 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:37.943120 17757 heartbeater.cc:511] Master 127.10.71.190:36521 requested a full tablet report, sending...
I20260619 14:57:37.943472 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:42443
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:36521
--webserver_port=41327
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:57:37.943585 17758 heartbeater.cc:347] Connected to a master server at 127.10.71.189:38191
I20260619 14:57:37.943651 17758 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:37.943813 17758 heartbeater.cc:511] Master 127.10.71.189:38191 requested a full tablet report, sending...
I20260619 14:57:37.944130 16649 ts_manager.cc:194] Re-registered known tserver with Master: 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
I20260619 14:57:37.944525 17247 ts_manager.cc:194] Registered new tserver with Master: 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
I20260619 14:57:37.944965 16649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:56465
I20260619 14:57:37.947753 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.929925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.10.71.130:52195)
0619 14:57:37.930145 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:37.930157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:37.930378 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:37.930800 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:37.930802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:37.931222 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:37.931562 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.931568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.932634 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.932637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.933335 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:37.933342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.933878 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.933880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:37.934005 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.934525 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:37.934539 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:37.943334 (+ 8795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:37.945251 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.945256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.945259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.945492 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.946713 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:37.946715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:37.946716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:37.946797 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:37.947237 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:37.947239 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:37.947338 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:57:37.947424 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:57:37.947582 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":157,"spinlock_wait_cycles":3840,"thread_start_us":79,"threads_started":1}
I20260619 14:57:37.948040 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:37.930033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:52195 (local address 127.10.71.188:42157)
0619 14:57:37.930140 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:37.930144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:37.930206 (+ 62us) server_negotiation.cc:407] Connection header received
0619 14:57:37.930535 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:37.930537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:37.930591 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:37.930648 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:37.931718 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.932493 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.933453 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:37.933754 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:37.933851 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:37.943448 (+ 9597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:37.943451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:37.943454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:37.943474 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:37.945155 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.946507 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.946512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.946517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.946594 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:37.946904 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:37.946909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:37.946913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:37.947142 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:37.947238 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:37.947501 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 14:57:37.947625 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:37.948577 17756 heartbeater.cc:347] Connected to a master server at 127.10.71.188:42157
I20260619 14:57:37.948673 17756 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:37.948817 17756 heartbeater.cc:511] Master 127.10.71.188:42157 requested a full tablet report, sending...
I20260619 14:57:37.949522 17440 ts_manager.cc:194] Registered new tserver with Master: 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
W20260619 14:57:38.041587 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:38.041779 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:38.041812 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:38.045568 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:38.045617 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:38.045718 17763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:38.050370 17763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:42443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=41327
--tserver_master_addrs=127.10.71.190:36521,127.10.71.189:38191,127.10.71.188:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:38.051133 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:38.051927 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:38.060135 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:38.060148 17771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:38.060133 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:38.060348 17763 server_base.cc:1034] running on GCE node
I20260619 14:57:38.060734 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:38.061291 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:38.062448 17763 hybrid_clock.cc:648] HybridClock initialized: now 1781881058062426 us; error 48 us; skew 500 ppm
Jun 19 14:57:38 dist-test-slave-3qdh krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881058, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:38.064966 17763 init.cc:377] Logged in from keytab as oryx/127.10.71.131@KRBTEST.COM (short username oryx)
I20260619 14:57:38.065704 17763 webserver.cc:460] Webserver started at http://127.10.71.131:41327/ using document root <none> and password file <none>
I20260619 14:57:38.066180 17763 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:38.066277 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:38.069815 17763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:57:38.071699 17778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:38.072839 17763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:38.072952 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "8e0b534270fb49cd8d5016202e853065"
format_stamp: "Formatted at 2026-06-19 14:57:36 on dist-test-slave-3qdh"
server_key: "d291b9e0199b21575a706b13fb5c4c01"
server_key_iv: "5003687ed4c60da8ea7ef930a134eab1"
server_key_version: "encryptionkey@0"
I20260619 14:57:38.073467 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:38.086139 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:38.088511 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:38.088675 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:38.089154 17763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:38.089861 17763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:38.089926 17763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:38.089986 17763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:38.090027 17763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:38.098037 17763 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:42443
I20260619 14:57:38.098058 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:42443 every 8 connection(s)
I20260619 14:57:38.098802 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:38.099578 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 17763
Jun 19 14:57:38 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881058, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.189@KRBTEST.COM
Jun 19 14:57:38 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881058, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.188@KRBTEST.COM
I20260619 14:57:38.114599 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.100655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:36705 (local address 127.10.71.188:42157)
0619 14:57:38.100760 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.100764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.101722 (+ 958us) server_negotiation.cc:407] Connection header received
0619 14:57:38.102392 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.102395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.102447 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.102499 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.104188 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.104984 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.106833 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.107160 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.107242 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.110597 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.110630 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.110633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.110659 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.112522 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.113005 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.113009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.113012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.113089 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.113370 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.113373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.113375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.113592 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.113673 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.114335 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.114444 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:57:38.114627 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.101771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:43387 (local address 127.10.71.189:38191)
0619 14:57:38.101886 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.101890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.102083 (+ 193us) server_negotiation.cc:407] Connection header received
0619 14:57:38.102397 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.102401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.102449 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.102497 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.104175 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.104997 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.106833 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.107167 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.107242 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.110055 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.110060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.110062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.110088 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.112247 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.112880 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.112883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.112885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.112994 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.113370 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.113373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.113375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.113608 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.113673 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.114342 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.114444 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:57:38.115319 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.103080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.10.71.189:38191)
0619 14:57:38.103329 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.103335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.105611 (+ 2276us) server_negotiation.cc:407] Connection header received
0619 14:57:38.105751 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.105754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.105794 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.105845 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.106703 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.107415 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.109365 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.109680 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.109753 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.111354 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.111357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.111359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.111380 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.113177 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.113673 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.113677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.113679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.113753 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.114368 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.114374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.114377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.114617 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.114726 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.115057 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.115162 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260619 14:57:38.115959 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.100811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.10.71.131:36705)
0619 14:57:38.101649 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.101660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.102183 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.102830 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.102838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.103456 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:38.104010 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.104029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.105161 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.105163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.106703 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.106713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.107408 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.107411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.108281 (+ 870us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.109099 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:38.109112 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:38.110435 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:38.112642 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:38.112646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:38.112649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:38.112877 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:38.113192 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:38.113194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:38.113196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:38.113266 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:38.113757 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:38.113763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:38.114067 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.115070 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.115165 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":1133,"spinlock_wait_cycles":4992,"thread_start_us":116,"threads_started":1}
I20260619 14:57:38.115959 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.101699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.10.71.131:43387)
0619 14:57:38.101986 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.101997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.102184 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.102830 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.102838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.103384 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:38.104011 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.104029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.105159 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.105162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.106703 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.106713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.107408 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.107411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.107605 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.108140 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:38.108163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:38.109893 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:38.112400 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:38.112407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:38.112420 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:38.112768 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:38.113098 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:38.113101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:38.113103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:38.113270 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:38.113757 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:38.113763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:38.114067 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.114816 (+ 749us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.115089 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":274,"spinlock_wait_cycles":2688,"thread_start_us":63,"threads_started":1}
Jun 19 14:57:38 dist-test-slave-3qdh krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881058, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:38.117461 17894 heartbeater.cc:347] Connected to a master server at 127.10.71.189:38191
I20260619 14:57:38.117663 17894 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:38.118085 17894 heartbeater.cc:511] Master 127.10.71.189:38191 requested a full tablet report, sending...
I20260619 14:57:38.118577 17892 heartbeater.cc:347] Connected to a master server at 127.10.71.188:42157
I20260619 14:57:38.118651 17892 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:38.118796 17892 heartbeater.cc:511] Master 127.10.71.188:42157 requested a full tablet report, sending...
I20260619 14:57:38.119052 17247 ts_manager.cc:194] Registered new tserver with Master: 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443)
I20260619 14:57:38.119434 17440 ts_manager.cc:194] Registered new tserver with Master: 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443)
I20260619 14:57:38.119882 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.102426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.10.71.190:36521)
0619 14:57:38.104359 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.104363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.104377 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:38.104414 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.104417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.104461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.105381 (+ 920us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.105479 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.106201 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.107355 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.107658 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.108096 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.109534 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.109537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.109540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.109562 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.112126 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.114463 (+ 2337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.114466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.114468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.115422 (+ 954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.115749 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.115755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.115759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.115971 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.119078 (+ 3107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.119153 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.119754 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1846,"thread_start_us":73,"threads_started":1}
I20260619 14:57:38.121440 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.100811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:54961)
0619 14:57:38.101292 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.101357 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.102184 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.102830 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.102838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.103738 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:38.104267 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.104273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.105285 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.105287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.108094 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.108105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.111023 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.111027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.113809 (+ 2782us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.116323 (+ 2514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:38.116342 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:38.117767 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:38.119513 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:38.119516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:38.119517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:38.119716 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:38.119999 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:38.120003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:38.120005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:38.120083 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:38.120959 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:38.120963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:38.121056 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.121156 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.121275 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":70,"thread_start_us":585,"threads_started":1}
I20260619 14:57:38.121440 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.100416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:54961 (local address 127.10.71.190:36521)
0619 14:57:38.100536 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.100540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.101468 (+ 928us) server_negotiation.cc:407] Connection header received
0619 14:57:38.102398 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.102401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.102449 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.102498 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.104414 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.105170 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.108517 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.108880 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.108961 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.117916 (+ 8955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.117919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.117921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.117941 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.119409 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.119825 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.119828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.119829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.119900 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.120593 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.120623 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.120625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.120809 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.120910 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.121222 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.121317 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:57:38.122362 17893 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36521
I20260619 14:57:38.122438 17893 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:38.122577 17893 heartbeater.cc:511] Master 127.10.71.190:36521 requested a full tablet report, sending...
I20260619 14:57:38.123116 16650 ts_manager.cc:194] Re-registered known tserver with Master: 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443)
I20260619 14:57:38.123728 16650 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:54961
I20260619 14:57:38.122761 16649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:57:38.125905 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.103839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.10.71.188:42157)
0619 14:57:38.104073 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.104085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.111168 (+ 7083us) server_negotiation.cc:407] Connection header received
0619 14:57:38.111209 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.111212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.111255 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.111331 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.112995 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.113811 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.120197 (+ 6386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.120531 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.120645 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.122533 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.122536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.122538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.122559 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.124202 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.124720 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.124723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.124725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.124813 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.125073 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.125077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.125078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.125251 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.125377 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.125663 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.125767 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
W20260619 14:57:38.126533 16649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:38.141928 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.136172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:46755 (local address 127.0.0.1:34564)
0619 14:57:38.136999 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.137009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.137107 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.137718 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.137720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.137735 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.137796 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.137800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.139003 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.139006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.140158 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.140167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.141552 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.141555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.141651 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.141673 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.141735 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.141796 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":65,"thread_start_us":86,"threads_started":1}
I20260619 14:57:38.142334 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.136358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.10.71.129:46755)
0619 14:57:38.136766 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.136778 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.137089 (+ 311us) server_negotiation.cc:407] Connection header received
0619 14:57:38.137258 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.137264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.137446 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.137559 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.137895 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.138875 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.140295 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.141259 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.141332 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.141460 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.142095 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.142202 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":95,"threads_started":1}
I20260619 14:57:38.142351 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.136771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:42443 (local address 127.0.0.1:37262)
0619 14:57:38.138281 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.138292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.138373 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.138724 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.138727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.138743 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.138837 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.138842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.139805 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.139808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.140954 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.140960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.141982 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.141988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.142089 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.142110 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.142158 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.142220 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":111,"threads_started":1}
I20260619 14:57:38.143000 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.137016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.10.71.131:42443)
0619 14:57:38.137268 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.137278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.138372 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 14:57:38.138463 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.138468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.138604 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.138733 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.138932 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.139682 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.141074 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.141861 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.142629 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.142755 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.142798 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.142873 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260619 14:57:38.143522 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.137063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.10.71.130:40103)
0619 14:57:38.137715 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.137727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.137745 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:57:38.137799 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.137803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.137928 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.138071 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.138294 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.139351 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.140892 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.141659 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.141756 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.141875 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.142556 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.142646 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":91,"threads_started":1}
I20260619 14:57:38.143724 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.136475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40103 (local address 127.0.0.1:60568)
0619 14:57:38.137379 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.137393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.137471 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.138060 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.138071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.138087 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.138165 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.138174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.139464 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.139468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.140762 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.140772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.141929 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.141932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.142123 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.142145 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.142210 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.142269 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":23,"thread_start_us":88,"threads_started":1}
I20260619 14:57:38.145071 17553 tablet_service.cc:1467] Processing CreateTablet for tablet 2c57115a574e4c4a8d16905bd3136fd9 (DEFAULT_TABLE table=test-table [id=3406c258525c470a8607644b3e7c55fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:38.145344 17826 tablet_service.cc:1467] Processing CreateTablet for tablet 2c57115a574e4c4a8d16905bd3136fd9 (DEFAULT_TABLE table=test-table [id=3406c258525c470a8607644b3e7c55fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:38.146116 17553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c57115a574e4c4a8d16905bd3136fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:38.146169 17826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c57115a574e4c4a8d16905bd3136fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:38.146234 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.141336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.0.0.1:58496)
0619 14:57:38.141807 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.141822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.141950 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.142558 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.142563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.142582 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.142661 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.142666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.143727 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.143731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.144922 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.144932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.145800 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.145804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.145916 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.145940 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.146047 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.146117 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":75,"threads_started":1}
I20260619 14:57:38.146255 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.141598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.10.71.188:42157)
0619 14:57:38.141694 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.141697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.141886 (+ 189us) server_negotiation.cc:407] Connection header received
0619 14:57:38.142194 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.142197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.142251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.142310 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.142782 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.143465 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.145057 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.145673 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.145788 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.145936 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.146069 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.146156 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:57:38.146728 17442 raft_consensus.cc:1203] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 14:57:38.148491 17922 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359478d2a1624cadbfdc4d89d8997014. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:38.148649 17922 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:57:38.150499 17690 tablet_service.cc:1467] Processing CreateTablet for tablet 2c57115a574e4c4a8d16905bd3136fd9 (DEFAULT_TABLE table=test-table [id=3406c258525c470a8607644b3e7c55fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:38.151192 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c57115a574e4c4a8d16905bd3136fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:38.153259 17926 tablet_bootstrap.cc:492] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Bootstrap starting.
I20260619 14:57:38.155095 17926 tablet_bootstrap.cc:654] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:38.155766 17926 log.cc:824] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:38.157542 17926 tablet_bootstrap.cc:492] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: No bootstrap required, opened a new log
I20260619 14:57:38.157750 17926 ts_tablet_manager.cc:1397] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:38.158416 17929 tablet_bootstrap.cc:492] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Bootstrap starting.
I20260619 14:57:38.159945 17929 tablet_bootstrap.cc:654] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:38.160593 17928 tablet_bootstrap.cc:492] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Bootstrap starting.
I20260619 14:57:38.160756 17929 log.cc:824] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:38.161161 17926 raft_consensus.cc:348] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.161419 17926 raft_consensus.cc:374] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:38.161481 17926 raft_consensus.cc:729] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65529b9919b04d4b923c9b3c52f0855d, State: Initialized, Role: FOLLOWER
I20260619 14:57:38.161913 17926 consensus_queue.cc:260] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.162498 17928 tablet_bootstrap.cc:654] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:38.162621 17929 tablet_bootstrap.cc:492] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: No bootstrap required, opened a new log
I20260619 14:57:38.162782 17929 ts_tablet_manager.cc:1397] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:38.162778 17926 ts_tablet_manager.cc:1428] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260619 14:57:38.163005 17620 heartbeater.cc:503] Master 127.10.71.190:36521 was elected leader, sending a full tablet report...
I20260619 14:57:38.163280 17928 log.cc:824] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:38.165271 17928 tablet_bootstrap.cc:492] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: No bootstrap required, opened a new log
I20260619 14:57:38.165419 17928 ts_tablet_manager.cc:1397] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:57:38.168735 17928 raft_consensus.cc:348] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.168941 17928 raft_consensus.cc:374] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:38.169018 17928 raft_consensus.cc:729] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0b534270fb49cd8d5016202e853065, State: Initialized, Role: FOLLOWER
I20260619 14:57:38.169224 17929 raft_consensus.cc:348] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.169447 17929 raft_consensus.cc:374] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:38.169508 17929 raft_consensus.cc:729] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eeb517267474aafaba6681953067269, State: Initialized, Role: FOLLOWER
I20260619 14:57:38.169445 17928 consensus_queue.cc:260] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.169962 17929 consensus_queue.cc:260] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.170396 17928 ts_tablet_manager.cc:1428] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:57:38.170571 17757 heartbeater.cc:503] Master 127.10.71.190:36521 was elected leader, sending a full tablet report...
I20260619 14:57:38.170603 17893 heartbeater.cc:503] Master 127.10.71.190:36521 was elected leader, sending a full tablet report...
I20260619 14:57:38.172637 17929 ts_tablet_manager.cc:1428] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260619 14:57:38.178637 17759 tablet.cc:2369] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:38.193852 17931 raft_consensus.cc:482] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:38.194025 17931 raft_consensus.cc:504] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.194898 17933 raft_consensus.cc:482] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:38.195024 17933 raft_consensus.cc:504] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.195511 17931 leader_election.cc:288] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eeb517267474aafaba6681953067269 (127.10.71.130:40103), 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443)
I20260619 14:57:38.195889 17933 leader_election.cc:288] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755), 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
I20260619 14:57:38.199486 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.195305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40103 (local address 127.10.71.129:57281)
0619 14:57:38.195454 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.195464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.195573 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.196073 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.196076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.196094 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.196221 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.196229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.197124 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.197127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.198212 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.198221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.199088 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.199091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.199196 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.199219 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.199278 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.199335 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":39}
I20260619 14:57:38.199887 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.196016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:46755 (local address 127.10.71.131:44977)
0619 14:57:38.196129 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.196140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.196186 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.196687 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.196690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.196704 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.196788 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.196792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.197636 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.197639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.198697 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.198706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.199493 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.199497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.199604 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.199622 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.199682 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.199746 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":2}
I20260619 14:57:38.199971 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.195345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:57281 (local address 127.10.71.130:40103)
0619 14:57:38.195439 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.195443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.195521 (+ 78us) server_negotiation.cc:407] Connection header received
0619 14:57:38.195664 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.195667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.195707 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.195774 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.196339 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.197018 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.198328 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.198907 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.198999 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.199042 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.199730 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.199874 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":33}
I20260619 14:57:38.199971 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.196375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:44977 (local address 127.10.71.129:46755)
0619 14:57:38.196457 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.196460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.196469 (+ 9us) server_negotiation.cc:407] Connection header received
0619 14:57:38.196498 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.196501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.196531 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.196588 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.196886 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.197524 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.198815 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.199361 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.199463 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.199529 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.199812 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.199878 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":37}
I20260619 14:57:38.200399 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.195814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:42443 (local address 127.10.71.129:45413)
0619 14:57:38.195935 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.195945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.196007 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.196270 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.196273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.196287 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.196418 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.196427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.197264 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.197267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.198338 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.198346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.199174 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.199177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.199417 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.200149 (+ 732us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.200206 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.200261 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":803}
I20260619 14:57:38.200372 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65529b9919b04d4b923c9b3c52f0855d" is_pre_election: true
I20260619 14:57:38.200376 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "65529b9919b04d4b923c9b3c52f0855d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eeb517267474aafaba6681953067269" is_pre_election: true
I20260619 14:57:38.200528 17573 raft_consensus.cc:2454] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0b534270fb49cd8d5016202e853065 in term 0.
I20260619 14:57:38.200556 17710 raft_consensus.cc:2454] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65529b9919b04d4b923c9b3c52f0855d in term 0.
I20260619 14:57:38.200955 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.195853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:45413 (local address 127.10.71.131:42443)
0619 14:57:38.195937 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.195942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.196036 (+ 94us) server_negotiation.cc:407] Connection header received
0619 14:57:38.196088 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.196091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.196126 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.196185 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.196524 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.197161 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.198443 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.198962 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.199054 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.199106 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.200738 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.200864 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:57:38.200984 17507 leader_election.cc:302] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65529b9919b04d4b923c9b3c52f0855d, 6eeb517267474aafaba6681953067269; no voters:
I20260619 14:57:38.201215 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.196210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40103 (local address 127.10.71.131:58021)
0619 14:57:38.196335 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:38.196345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:38.196400 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:38.197023 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:38.197026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:38.197039 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:38.197127 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.197131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.198292 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.198295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.199389 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:38.199398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.200875 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.200878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:38.200977 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.200999 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:38.201046 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:57:38.201118 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":32}
I20260619 14:57:38.201226 17931 raft_consensus.cc:2781] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:38.201361 17846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "65529b9919b04d4b923c9b3c52f0855d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0b534270fb49cd8d5016202e853065" is_pre_election: true
I20260619 14:57:38.201373 17780 leader_election.cc:302] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65529b9919b04d4b923c9b3c52f0855d, 8e0b534270fb49cd8d5016202e853065; no voters:
I20260619 14:57:38.201531 17931 raft_consensus.cc:482] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:38.201547 17846 raft_consensus.cc:2454] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65529b9919b04d4b923c9b3c52f0855d in term 0.
I20260619 14:57:38.201630 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.196582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:58021 (local address 127.10.71.130:40103)
0619 14:57:38.196830 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.196837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.196849 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:38.196883 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.196886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.196918 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.197215 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:38.197313 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.198184 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.199519 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.200748 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.201381 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.201427 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.201459 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.201512 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260619 14:57:38.201627 17931 raft_consensus.cc:3037] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:38.201745 17933 raft_consensus.cc:2781] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:38.201812 17933 raft_consensus.cc:482] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:38.201866 17933 raft_consensus.cc:3037] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:38.201875 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eeb517267474aafaba6681953067269" is_pre_election: true
I20260619 14:57:38.201985 17710 raft_consensus.cc:2454] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0b534270fb49cd8d5016202e853065 in term 0.
I20260619 14:57:38.202975 17931 raft_consensus.cc:504] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.202975 17933 raft_consensus.cc:504] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.203246 17933 leader_election.cc:288] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 election: Requested vote from peers 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755), 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
I20260619 14:57:38.203248 17931 leader_election.cc:288] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [CANDIDATE]: Term 1 election: Requested vote from peers 6eeb517267474aafaba6681953067269 (127.10.71.130:40103), 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443)
I20260619 14:57:38.203608 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65529b9919b04d4b923c9b3c52f0855d"
I20260619 14:57:38.203620 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "65529b9919b04d4b923c9b3c52f0855d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eeb517267474aafaba6681953067269"
I20260619 14:57:38.203598 17846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "65529b9919b04d4b923c9b3c52f0855d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0b534270fb49cd8d5016202e853065"
I20260619 14:57:38.203773 17573 raft_consensus.cc:2379] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e0b534270fb49cd8d5016202e853065 in current term 1: Already voted for candidate 65529b9919b04d4b923c9b3c52f0855d in this term.
I20260619 14:57:38.203714 17710 raft_consensus.cc:3037] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:38.203797 17846 raft_consensus.cc:2379] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65529b9919b04d4b923c9b3c52f0855d in current term 1: Already voted for candidate 8e0b534270fb49cd8d5016202e853065 in this term.
I20260619 14:57:38.203711 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c57115a574e4c4a8d16905bd3136fd9" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eeb517267474aafaba6681953067269"
I20260619 14:57:38.204228 17780 leader_election.cc:302] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0b534270fb49cd8d5016202e853065; no voters: 65529b9919b04d4b923c9b3c52f0855d, 6eeb517267474aafaba6681953067269
I20260619 14:57:38.204430 17933 raft_consensus.cc:2726] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 14:57:38.204936 17710 raft_consensus.cc:2454] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65529b9919b04d4b923c9b3c52f0855d in term 1.
I20260619 14:57:38.205217 17507 leader_election.cc:302] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65529b9919b04d4b923c9b3c52f0855d, 6eeb517267474aafaba6681953067269; no voters: 8e0b534270fb49cd8d5016202e853065
I20260619 14:57:38.205381 17931 raft_consensus.cc:2781] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:38.205570 17931 raft_consensus.cc:686] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 1 LEADER]: Becoming Leader. State: Replica: 65529b9919b04d4b923c9b3c52f0855d, State: Running, Role: LEADER
I20260619 14:57:38.205842 17931 consensus_queue.cc:237] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:38.208447 16650 catalog_manager.cc:5495] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d reported cstate change: term changed from 0 to 1, leader changed from <none> to 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129). New cstate: current_term: 1 leader_uuid: "65529b9919b04d4b923c9b3c52f0855d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:38.209764 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: NON_VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: true } } }
I20260619 14:57:38.209872 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:38.211263 17404 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: attempting to promote NON_VOTER de3417a8d9ef473daf305799cc15e3e2 to VOTER
I20260619 14:57:38.211534 17404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:38.212775 17257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Refusing update from remote peer 359478d2a1624cadbfdc4d89d8997014: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 14:57:38.213090 17442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 LEARNER]: Refusing update from remote peer 359478d2a1624cadbfdc4d89d8997014: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 14:57:38.213132 17910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 14:57:38.213405 17910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 14:57:38.214480 17914 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.214980 17257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.215457 17442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.215989 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.216069 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:38.216168 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.216223 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:38.216353 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.216423 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:38.216881 17922 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.216886 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359478d2a1624cadbfdc4d89d8997014" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:38.216986 17922 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:38.217080 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:38.239090 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.235515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.10.71.129:46755)
0619 14:57:38.235649 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.235652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.235665 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:38.235760 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.235762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.235807 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.235903 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:38.236130 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.236708 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.237576 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.237803 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.237915 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.238237 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:38.238683 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:38.238837 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.238874 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.238925 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:57:38.239912 17622 tablet.cc:2369] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:38.244216 17846 raft_consensus.cc:1261] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Refusing update from remote peer 65529b9919b04d4b923c9b3c52f0855d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:38.244216 17710 raft_consensus.cc:1261] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Refusing update from remote peer 65529b9919b04d4b923c9b3c52f0855d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:38.244837 17936 consensus_queue.cc:1035] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:38.245023 17931 consensus_queue.cc:1035] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:38.264215 16649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:38.264376 16649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:38.268831 16649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 359478d2a1624cadbfdc4d89d8997014: Sending DeleteTablet for 3 replicas of tablet 2c57115a574e4c4a8d16905bd3136fd9
I20260619 14:57:38.269536 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c57115a574e4c4a8d16905bd3136fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:38 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:60568
I20260619 14:57:38.269539 17826 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c57115a574e4c4a8d16905bd3136fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:38 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:37262
I20260619 14:57:38.269623 17553 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c57115a574e4c4a8d16905bd3136fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:38 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34564
I20260619 14:57:38.269989 17952 tablet_replica.cc:330] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: stopping tablet replica
I20260619 14:57:38.270292 17952 raft_consensus.cc:2229] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:38.270608 17952 raft_consensus.cc:2258] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:38.270957 17951 tablet_replica.cc:330] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: stopping tablet replica
I20260619 14:57:38.271219 17951 raft_consensus.cc:2229] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:38.271389 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 16618
I20260619 14:57:38.271456 17951 raft_consensus.cc:2258] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:38.273208 17952 ts_tablet_manager.cc:1905] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:38.278971 17952 ts_tablet_manager.cc:1918] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:38.279059 17952 log.cc:1196] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/2c57115a574e4c4a8d16905bd3136fd9
I20260619 14:57:38.279331 17952 ts_tablet_manager.cc:1939] T 2c57115a574e4c4a8d16905bd3136fd9 P 6eeb517267474aafaba6681953067269: Deleting consensus metadata
I20260619 14:57:38.279713 17951 ts_tablet_manager.cc:1905] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:38.280402 17953 tablet_replica.cc:330] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: stopping tablet replica
I20260619 14:57:38.280656 17953 raft_consensus.cc:2229] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:38.280987 17953 raft_consensus.cc:2258] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:38.282176 17953 ts_tablet_manager.cc:1905] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:38.282377 17951 ts_tablet_manager.cc:1918] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:38.282450 17951 log.cc:1196] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/2c57115a574e4c4a8d16905bd3136fd9
I20260619 14:57:38.282521 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.271122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.10.71.188:42157)
0619 14:57:38.272661 (+ 1539us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.272668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.272871 (+ 203us) server_negotiation.cc:407] Connection header received
0619 14:57:38.272964 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.272969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.273019 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.273101 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.273984 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.274870 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.275873 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.276316 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.276452 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.278501 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.278504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.278507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.278531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.280200 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.281166 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.281173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.281177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.281260 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.281575 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.281578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.281580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.281765 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.281860 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.282247 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.282360 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337}
I20260619 14:57:38.282773 17951 ts_tablet_manager.cc:1939] T 2c57115a574e4c4a8d16905bd3136fd9 P 8e0b534270fb49cd8d5016202e853065: Deleting consensus metadata
W20260619 14:57:38.283996 17782 connection.cc:535] server connection from 127.0.0.1:37262 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 14:57:38.284018 17643 connection.cc:535] server connection from 127.0.0.1:60568 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:57:38.284660 17953 ts_tablet_manager.cc:1918] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:38.284757 17953 log.cc:1196] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/2c57115a574e4c4a8d16905bd3136fd9
I20260619 14:57:38.285027 17953 ts_tablet_manager.cc:1939] T 2c57115a574e4c4a8d16905bd3136fd9 P 65529b9919b04d4b923c9b3c52f0855d: Deleting consensus metadata
W20260619 14:57:38.285794 17509 connection.cc:441] server connection from 127.0.0.1:34564 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34564 (request call id 1) could send its response
I20260619 14:57:38.295154 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.286933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.10.71.189:38191)
0619 14:57:38.287040 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.287044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.287242 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:57:38.287452 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.287455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.287502 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.287583 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.288495 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.289213 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.289968 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.290328 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.290433 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.291962 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.291965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.291968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.291988 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.293566 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.293964 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.293970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.293971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.294054 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.294324 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.294327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.294329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.294495 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.294643 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.294867 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.294980 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:38.295922 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:38.287267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.10.71.188:42157)
0619 14:57:38.287362 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:38.287366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:38.287666 (+ 300us) server_negotiation.cc:407] Connection header received
0619 14:57:38.287765 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:38.287784 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:38.287831 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:38.287911 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:38.289034 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.289721 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.290606 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:38.290936 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:38.291019 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:38.292655 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:38.292659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:38.292661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:38.292684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:38.294230 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.294706 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.294709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.294711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.294785 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:38.295049 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:38.295052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:38.295054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:38.295243 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:38.295332 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:38.295670 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 14:57:38.295777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:57:39.260860 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.260358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:47983)
0619 14:57:39.260625 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.260702 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
W20260619 14:57:39.261376 17893 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:36521 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.261536 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.261084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:60599)
0619 14:57:39.261354 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.261442 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":62,"threads_started":1}
W20260619 14:57:39.261795 17620 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:36521 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.262341 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.261364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:54855)
0619 14:57:39.262166 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.262243 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":59,"threads_started":1}
W20260619 14:57:39.262557 17757 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:36521 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.769284 17964 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 359478d2a1624cadbfdc4d89d8997014)
I20260619 14:57:39.769500 17964 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:39.769841 17965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 359478d2a1624cadbfdc4d89d8997014)
I20260619 14:57:39.770031 17965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:39.770581 17964 leader_election.cc:288] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521), 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189:38191)
I20260619 14:57:39.770953 17965 leader_election.cc:288] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521), de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188:42157)
I20260619 14:57:39.771214 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.770710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52658)
0619 14:57:39.771016 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.771100 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":66,"threads_started":1}
W20260619 14:57:39.771366 17426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.771680 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.771090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52674)
0619 14:57:39.771523 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.771595 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":60,"threads_started":1}
W20260619 14:57:39.771826 17233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.772279 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.772043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52686)
0619 14:57:39.772171 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.772216 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 14:57:39.772433 17426 leader_election.cc:334] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.772822 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.772473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52692)
0619 14:57:39.772725 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.772769 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260619 14:57:39.772949 17233 leader_election.cc:334] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.775168 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.771308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.10.71.188:42157)
0619 14:57:39.771621 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:39.771627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:39.771956 (+ 329us) server_negotiation.cc:407] Connection header received
0619 14:57:39.772000 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:39.772002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:39.772042 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:39.772111 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:39.772497 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.773159 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.774179 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.774579 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.774696 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:39.774753 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:39.775003 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 14:57:39.775063 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":50,"threads_started":1}
I20260619 14:57:39.775218 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.770758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:38191 (local address 127.0.0.1:44442)
0619 14:57:39.771210 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.771224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:39.771431 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:39.772137 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:39.772140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:39.772169 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:39.772270 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:39.772277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.773400 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.773405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:39.774171 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:39.774180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.774822 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.774825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:39.774932 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:39.774951 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:57:39.775023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:57:39.775097 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":1920,"thread_start_us":55,"threads_started":1}
I20260619 14:57:39.775202 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.771340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:42157 (local address 127.0.0.1:58524)
0619 14:57:39.771739 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.771755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:39.771871 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:39.772217 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:39.772220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:39.772236 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:39.772370 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:39.772378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.773276 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.773279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:39.774057 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:39.774065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.774696 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.774700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:39.774814 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:39.774890 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:57:39.774979 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:57:39.775051 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":40,"spinlock_wait_cycles":10240,"thread_start_us":81,"threads_started":1}
I20260619 14:57:39.775218 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.770883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.10.71.189:38191)
0619 14:57:39.771661 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:39.771665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:39.771677 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:39.771817 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:39.771820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:39.771930 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:39.772012 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:39.772620 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.773289 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.774290 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:39.774697 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:39.774842 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:39.774956 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:39.775036 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 14:57:39.775097 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":704,"spinlock_wait_cycles":10368,"thread_start_us":47,"threads_started":1}
I20260619 14:57:39.775808 17442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e745a031fe4a348f1063ffd5df6651" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de3417a8d9ef473daf305799cc15e3e2" is_pre_election: true
I20260619 14:57:39.775871 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de3417a8d9ef473daf305799cc15e3e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16e745a031fe4a348f1063ffd5df6651" is_pre_election: true
I20260619 14:57:39.776001 17442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16e745a031fe4a348f1063ffd5df6651 in term 1.
I20260619 14:57:39.776001 17257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de3417a8d9ef473daf305799cc15e3e2 in term 1.
I20260619 14:57:39.776376 17234 leader_election.cc:302] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16e745a031fe4a348f1063ffd5df6651, de3417a8d9ef473daf305799cc15e3e2; no voters: 359478d2a1624cadbfdc4d89d8997014
I20260619 14:57:39.776422 17427 leader_election.cc:302] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16e745a031fe4a348f1063ffd5df6651, de3417a8d9ef473daf305799cc15e3e2; no voters: 359478d2a1624cadbfdc4d89d8997014
I20260619 14:57:39.776566 17965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:57:39.776621 17965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 359478d2a1624cadbfdc4d89d8997014)
I20260619 14:57:39.776659 17965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:57:39.776738 17964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:57:39.776795 17964 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 359478d2a1624cadbfdc4d89d8997014)
I20260619 14:57:39.776857 17964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:57:39.777915 17965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:39.777915 17964 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:39.778249 17964 leader_election.cc:288] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [CANDIDATE]: Term 2 election: Requested vote from peers 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521), 16e745a031fe4a348f1063ffd5df6651 (127.10.71.189:38191)
I20260619 14:57:39.778249 17965 leader_election.cc:288] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 2 election: Requested vote from peers 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521), de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188:42157)
I20260619 14:57:39.778537 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.778341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52704)
0619 14:57:39.778430 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.778481 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 14:57:39.778685 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.778341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52706)
0619 14:57:39.778504 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.778548 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 14:57:39.778659 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de3417a8d9ef473daf305799cc15e3e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16e745a031fe4a348f1063ffd5df6651"
I20260619 14:57:39.778774 17442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e745a031fe4a348f1063ffd5df6651" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de3417a8d9ef473daf305799cc15e3e2"
I20260619 14:57:39.778859 17257 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate de3417a8d9ef473daf305799cc15e3e2 in current term 2: Already voted for candidate 16e745a031fe4a348f1063ffd5df6651 in this term.
I20260619 14:57:39.778910 17442 raft_consensus.cc:2379] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 16e745a031fe4a348f1063ffd5df6651 in current term 2: Already voted for candidate de3417a8d9ef473daf305799cc15e3e2 in this term.
I20260619 14:57:39.779392 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.779124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52724)
0619 14:57:39.779285 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.779335 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 14:57:39.779454 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:39.779249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52728)
0619 14:57:39.779356 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:39.779396 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:57:39.779470 17426 leader_election.cc:334] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
W20260619 14:57:39.779515 17233 leader_election.cc:334] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:39.779531 17426 leader_election.cc:302] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de3417a8d9ef473daf305799cc15e3e2; no voters: 16e745a031fe4a348f1063ffd5df6651, 359478d2a1624cadbfdc4d89d8997014
I20260619 14:57:39.779592 17233 leader_election.cc:302] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16e745a031fe4a348f1063ffd5df6651; no voters: 359478d2a1624cadbfdc4d89d8997014, de3417a8d9ef473daf305799cc15e3e2
I20260619 14:57:39.779680 17964 raft_consensus.cc:2726] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:57:39.779779 17965 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:57:40.262188 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:40.261809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:56631)
0619 14:57:40.262030 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:40.262088 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260619 14:57:40.262748 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:40.262224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:60119)
0619 14:57:40.262610 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:40.262662 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":73,"threads_started":1}
I20260619 14:57:40.263414 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:40.263056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:35849)
0619 14:57:40.263266 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:40.263336 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":57,"threads_started":1}
I20260619 14:57:41.263418 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.263037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:49551)
0619 14:57:41.263261 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.263317 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260619 14:57:41.264086 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.263370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:37559)
0619 14:57:41.263924 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.263982 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":54,"threads_started":1}
I20260619 14:57:41.264369 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.263969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:33771)
0619 14:57:41.264192 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.264252 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260619 14:57:41.364820 17980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:41.364949 17980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:41.365267 17980 leader_election.cc:288] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521), de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188:42157)
I20260619 14:57:41.365653 17442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e745a031fe4a348f1063ffd5df6651" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de3417a8d9ef473daf305799cc15e3e2" is_pre_election: true
I20260619 14:57:41.365818 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.365428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52948)
0619 14:57:41.365663 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.365725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260619 14:57:41.365803 17442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16e745a031fe4a348f1063ffd5df6651 in term 2.
I20260619 14:57:41.366186 17234 leader_election.cc:302] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e745a031fe4a348f1063ffd5df6651, de3417a8d9ef473daf305799cc15e3e2; no voters:
I20260619 14:57:41.366357 17980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 14:57:41.366439 17980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:41.366521 17980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:57:41.366770 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.366560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52954)
0619 14:57:41.366647 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.366719 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 14:57:41.366873 17233 leader_election.cc:334] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:41.367717 17980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:41.368093 17980 leader_election.cc:288] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 3 election: Requested vote from peers 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521), de3417a8d9ef473daf305799cc15e3e2 (127.10.71.188:42157)
I20260619 14:57:41.368484 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.368204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52964)
0619 14:57:41.368380 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.368436 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:57:41.368433 17442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e745a031fe4a348f1063ffd5df6651" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de3417a8d9ef473daf305799cc15e3e2"
I20260619 14:57:41.368595 17442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:57:41.369165 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.368978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52966)
0619 14:57:41.369074 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.369120 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:57:41.369334 17233 leader_election.cc:334] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
I20260619 14:57:41.369765 17442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16e745a031fe4a348f1063ffd5df6651 in term 3.
I20260619 14:57:41.370041 17234 leader_election.cc:302] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16e745a031fe4a348f1063ffd5df6651, de3417a8d9ef473daf305799cc15e3e2; no voters: 359478d2a1624cadbfdc4d89d8997014
I20260619 14:57:41.370455 17980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:57:41.370676 17980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [term 3 LEADER]: Becoming Leader. State: Replica: 16e745a031fe4a348f1063ffd5df6651, State: Running, Role: LEADER
I20260619 14:57:41.370922 17980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } }
I20260619 14:57:41.371986 17983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e745a031fe4a348f1063ffd5df6651. Latest consensus state: current_term: 3 leader_uuid: "16e745a031fe4a348f1063ffd5df6651" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:41.372103 17983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:41.372433 17985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:41.374389 17985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:41.374778 17985 catalog_manager.cc:1259] Loaded cluster ID: 262ce88f198349a4aae271a44ca707fc
I20260619 14:57:41.374845 17985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:41.375371 17985 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:41.375730 17985 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Loaded TSK: 0
I20260619 14:57:41.376444 17985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:57:41.400980 17247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:41.403617 17247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:41.406105 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.405922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52980)
0619 14:57:41.406017 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.406061 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:57:41.406242 17442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [term 3 FOLLOWER]: Refusing update from remote peer 16e745a031fe4a348f1063ffd5df6651: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 14:57:41.406711 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.406498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.0.0.1:52988)
0619 14:57:41.406583 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.406625 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 14:57:41.406632 17983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 14:57:41.406823 17233 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 -> Peer 359478d2a1624cadbfdc4d89d8997014 (127.10.71.190:36521): Couldn't send request to peer 359478d2a1624cadbfdc4d89d8997014. Status: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:57:41.407907 17987 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e745a031fe4a348f1063ffd5df6651. Latest consensus state: current_term: 3 leader_uuid: "16e745a031fe4a348f1063ffd5df6651" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:41.408008 17987 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:41.408949 17983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "16e745a031fe4a348f1063ffd5df6651" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:41.409071 17983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:41.409319 17987 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "16e745a031fe4a348f1063ffd5df6651" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "359478d2a1624cadbfdc4d89d8997014" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36521 } } peers { permanent_uuid: "16e745a031fe4a348f1063ffd5df6651" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 38191 } attrs { promote: false } } peers { permanent_uuid: "de3417a8d9ef473daf305799cc15e3e2" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 42157 } attrs { promote: false } } }
I20260619 14:57:41.409417 17987 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3417a8d9ef473daf305799cc15e3e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:57:41.418814 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.414487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:46755 (local address 127.0.0.1:34578)
0619 14:57:41.414587 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.414599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:41.415355 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:41.415751 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:41.415754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:41.415769 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:41.415886 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:41.415892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.416896 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.416899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:41.417738 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:41.417750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.418504 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:41.418627 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.418648 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:57:41.418704 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:57:41.418746 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":16}
I20260619 14:57:41.418910 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.415080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.10.71.129:46755)
0619 14:57:41.415497 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:41.415500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:41.415513 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:41.415581 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:41.415584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:41.415626 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:41.415735 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:41.416016 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.416764 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.417885 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.418394 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418594 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.418647 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:41.418715 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:57:41.418794 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":57,"threads_started":1}
I20260619 14:57:41.419312 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.414725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40103 (local address 127.0.0.1:60572)
0619 14:57:41.415576 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.415589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:41.415699 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:41.415925 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:41.415927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:41.415941 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:41.416026 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:41.416031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.416854 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.416857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:41.417811 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:41.417819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.418500 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:41.419073 (+ 570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.419093 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:57:41.419149 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:57:41.419208 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":36,"thread_start_us":97,"threads_started":1}
I20260619 14:57:41.419332 17553 tablet_service.cc:1467] Processing CreateTablet for tablet 395959c3018e4c05801479efa7efc202 (DEFAULT_TABLE table=test-table [id=f28e44b750fa4062a53954370121d8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:41.419450 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.415057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:42443 (local address 127.0.0.1:37268)
0619 14:57:41.415925 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.415938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:41.416015 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:41.416273 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:41.416276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:41.416313 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:41.416387 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:41.416391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.417221 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.417224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:41.418152 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:41.418159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.418912 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:41.419188 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.419210 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:41.419273 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:57:41.419340 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":71,"threads_started":1}
I20260619 14:57:41.419548 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.414862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.10.71.130:40103)
0619 14:57:41.415174 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:41.415178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:41.415672 (+ 494us) server_negotiation.cc:407] Connection header received
0619 14:57:41.415771 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:41.415774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:41.415811 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:41.415919 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:41.416128 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.416743 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.417930 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.418402 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418504 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.418561 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:41.419178 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 14:57:41.419242 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260619 14:57:41.419585 17553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395959c3018e4c05801479efa7efc202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:41.419786 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.415256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.10.71.131:42443)
0619 14:57:41.415459 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:41.415463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:41.416083 (+ 620us) server_negotiation.cc:407] Connection header received
0619 14:57:41.416135 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:41.416137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:41.416179 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:41.416255 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:41.416491 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.417125 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418237 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.418732 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.418814 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.418863 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:41.419618 (+ 755us) server_negotiation.cc:299] Negotiation successful
0619 14:57:41.419683 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260619 14:57:41.419961 17690 tablet_service.cc:1467] Processing CreateTablet for tablet 395959c3018e4c05801479efa7efc202 (DEFAULT_TABLE table=test-table [id=f28e44b750fa4062a53954370121d8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:41.420202 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395959c3018e4c05801479efa7efc202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:41.420344 17826 tablet_service.cc:1467] Processing CreateTablet for tablet 395959c3018e4c05801479efa7efc202 (DEFAULT_TABLE table=test-table [id=f28e44b750fa4062a53954370121d8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:41.420552 17826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395959c3018e4c05801479efa7efc202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:41.422430 17999 tablet_bootstrap.cc:492] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: Bootstrap starting.
I20260619 14:57:41.422719 18000 tablet_bootstrap.cc:492] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Bootstrap starting.
I20260619 14:57:41.423007 17998 tablet_bootstrap.cc:492] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d: Bootstrap starting.
I20260619 14:57:41.423171 17999 tablet_bootstrap.cc:654] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:41.423472 18000 tablet_bootstrap.cc:654] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:41.423871 17998 tablet_bootstrap.cc:654] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:41.424347 17999 tablet_bootstrap.cc:492] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: No bootstrap required, opened a new log
I20260619 14:57:41.424453 17999 ts_tablet_manager.cc:1397] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:41.424712 18000 tablet_bootstrap.cc:492] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: No bootstrap required, opened a new log
I20260619 14:57:41.424810 18000 ts_tablet_manager.cc:1397] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:41.425210 17998 tablet_bootstrap.cc:492] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d: No bootstrap required, opened a new log
I20260619 14:57:41.425204 18000 raft_consensus.cc:348] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.425290 17998 ts_tablet_manager.cc:1397] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:41.425315 18000 raft_consensus.cc:374] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:41.425349 18000 raft_consensus.cc:729] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0b534270fb49cd8d5016202e853065, State: Initialized, Role: FOLLOWER
I20260619 14:57:41.425467 18000 consensus_queue.cc:260] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.425428 17999 raft_consensus.cc:348] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.425559 17999 raft_consensus.cc:374] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:41.425595 17999 raft_consensus.cc:729] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eeb517267474aafaba6681953067269, State: Initialized, Role: FOLLOWER
I20260619 14:57:41.425657 17998 raft_consensus.cc:348] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.425748 17998 raft_consensus.cc:374] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:41.425720 17999 consensus_queue.cc:260] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.425791 17998 raft_consensus.cc:729] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65529b9919b04d4b923c9b3c52f0855d, State: Initialized, Role: FOLLOWER
I20260619 14:57:41.425807 18000 ts_tablet_manager.cc:1428] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:41.425901 17998 consensus_queue.cc:260] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.426124 17999 ts_tablet_manager.cc:1428] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:41.426205 17998 ts_tablet_manager.cc:1428] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:41.426378 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.426128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:35289)
0619 14:57:41.426263 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.426306 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:57:41.426885 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.426652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:34535)
0619 14:57:41.426785 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.426838 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 14:57:41.428048 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.427823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:49727)
0619 14:57:41.427934 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.427988 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:57:41.429522 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.429299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:42873)
0619 14:57:41.429397 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.429461 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:57:41.430425 17894 heartbeater.cc:503] Master 127.10.71.189:38191 was elected leader, sending a full tablet report...
I20260619 14:57:41.430706 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.430491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:42669)
0619 14:57:41.430608 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.430651 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:57:41.431003 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.430752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:35073)
0619 14:57:41.430902 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.430950 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 14:57:41.431663 17621 heartbeater.cc:503] Master 127.10.71.189:38191 was elected leader, sending a full tablet report...
W20260619 14:57:41.602161 17895 tablet.cc:2369] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:41.722415 18004 raft_consensus.cc:482] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:41.722537 18004 raft_consensus.cc:504] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.722903 18004 leader_election.cc:288] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755), 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
I20260619 14:57:41.723475 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395959c3018e4c05801479efa7efc202" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eeb517267474aafaba6681953067269" is_pre_election: true
I20260619 14:57:41.723474 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395959c3018e4c05801479efa7efc202" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65529b9919b04d4b923c9b3c52f0855d" is_pre_election: true
I20260619 14:57:41.723631 17573 raft_consensus.cc:2454] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0b534270fb49cd8d5016202e853065 in term 0.
I20260619 14:57:41.723630 17710 raft_consensus.cc:2454] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0b534270fb49cd8d5016202e853065 in term 0.
I20260619 14:57:41.724025 17780 leader_election.cc:302] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65529b9919b04d4b923c9b3c52f0855d, 8e0b534270fb49cd8d5016202e853065; no voters:
I20260619 14:57:41.724231 18004 raft_consensus.cc:2781] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:41.724349 18004 raft_consensus.cc:482] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:41.724419 18004 raft_consensus.cc:3037] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:41.725521 18004 raft_consensus.cc:504] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.725821 18004 leader_election.cc:288] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 election: Requested vote from peers 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755), 6eeb517267474aafaba6681953067269 (127.10.71.130:40103)
I20260619 14:57:41.726130 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395959c3018e4c05801479efa7efc202" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65529b9919b04d4b923c9b3c52f0855d"
I20260619 14:57:41.726264 17573 raft_consensus.cc:3037] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:41.726291 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395959c3018e4c05801479efa7efc202" candidate_uuid: "8e0b534270fb49cd8d5016202e853065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eeb517267474aafaba6681953067269"
I20260619 14:57:41.726442 17710 raft_consensus.cc:3037] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:41.727166 17573 raft_consensus.cc:2454] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0b534270fb49cd8d5016202e853065 in term 1.
I20260619 14:57:41.727388 17710 raft_consensus.cc:2454] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0b534270fb49cd8d5016202e853065 in term 1.
I20260619 14:57:41.727437 17780 leader_election.cc:302] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65529b9919b04d4b923c9b3c52f0855d, 8e0b534270fb49cd8d5016202e853065; no voters:
I20260619 14:57:41.727685 18004 raft_consensus.cc:2781] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:41.727886 18004 raft_consensus.cc:686] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0b534270fb49cd8d5016202e853065, State: Running, Role: LEADER
I20260619 14:57:41.728202 18004 consensus_queue.cc:237] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } }
I20260619 14:57:41.728559 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.728295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:58215)
0619 14:57:41.728455 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.728500 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 14:57:41.730361 17246 catalog_manager.cc:5495] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0b534270fb49cd8d5016202e853065 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "8e0b534270fb49cd8d5016202e853065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0b534270fb49cd8d5016202e853065" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42443 } health_report { overall_health: HEALTHY } } }
I20260619 14:57:41.774904 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.771714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.10.71.131:42443)
0619 14:57:41.771840 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:41.771845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:41.771857 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:41.771972 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:41.771974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:41.772019 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:41.772122 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:41.772351 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.772790 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.773529 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:41.773723 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:41.773862 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:41.774123 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:41.774510 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:41.774636 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:41.774697 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 14:57:41.774746 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:41.777374 17710 raft_consensus.cc:1261] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Refusing update from remote peer 8e0b534270fb49cd8d5016202e853065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:41.777534 17573 raft_consensus.cc:1261] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Refusing update from remote peer 8e0b534270fb49cd8d5016202e853065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:41.777750 18007 consensus_queue.cc:1035] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eeb517267474aafaba6681953067269" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:41.777948 18004 consensus_queue.cc:1035] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65529b9919b04d4b923c9b3c52f0855d" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:41.779012 17758 heartbeater.cc:503] Master 127.10.71.189:38191 was elected leader, sending a full tablet report...
I20260619 14:57:41.779572 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.779345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:54737)
0619 14:57:41.779463 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.779514 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:57:41.780179 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.779915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:55755)
0619 14:57:41.780074 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.780121 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 14:57:41.781881 18010 mvcc.cc:205] Tried to move back new op lower bound from 7298584829035470848 to 7298584828841361408. Current Snapshot: MvccSnapshot[applied={T|T < 7298584829035470848}]
I20260619 14:57:41.782302 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.782092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:38161)
0619 14:57:41.782207 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.782249 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 14:57:41.782727 18012 mvcc.cc:205] Tried to move back new op lower bound from 7298584829035470848 to 7298584828841361408. Current Snapshot: MvccSnapshot[applied={T|T < 7298584829035470848 or (T in {7298584829035470848})}]
I20260619 14:57:41.783142 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.782909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.129:48953)
0619 14:57:41.783027 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.783071 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:57:41.786489 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.786222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.130:46047)
0619 14:57:41.786378 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.786427 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 14:57:41.786743 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.785474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36521 (local address 127.10.71.131:56003)
0619 14:57:41.786651 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.786695 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1128}
I20260619 14:57:41.787209 18011 mvcc.cc:205] Tried to move back new op lower bound from 7298584829035470848 to 7298584828841361408. Current Snapshot: MvccSnapshot[applied={T|T < 7298584829035470848 or (T in {7298584829035470848})}]
I20260619 14:57:41.789671 17247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:41.789804 17247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:41.793067 17247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16e745a031fe4a348f1063ffd5df6651: Sending DeleteTablet for 3 replicas of tablet 395959c3018e4c05801479efa7efc202
I20260619 14:57:41.793778 17553 tablet_service.cc:1514] Processing DeleteTablet for tablet 395959c3018e4c05801479efa7efc202 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:41 UTC) from {username='oryx', principal='oryx/127.10.71.189@KRBTEST.COM'} at 127.0.0.1:34578
I20260619 14:57:41.793923 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet 395959c3018e4c05801479efa7efc202 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:41 UTC) from {username='oryx', principal='oryx/127.10.71.189@KRBTEST.COM'} at 127.0.0.1:60572
I20260619 14:57:41.794101 18020 tablet_replica.cc:330] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d: stopping tablet replica
I20260619 14:57:41.794098 17826 tablet_service.cc:1514] Processing DeleteTablet for tablet 395959c3018e4c05801479efa7efc202 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:41 UTC) from {username='oryx', principal='oryx/127.10.71.189@KRBTEST.COM'} at 127.0.0.1:37268
I20260619 14:57:41.794224 18020 raft_consensus.cc:2229] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:41.794389 18020 raft_consensus.cc:2258] T 395959c3018e4c05801479efa7efc202 P 65529b9919b04d4b923c9b3c52f0855d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:41.794492 18022 tablet_replica.cc:330] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: stopping tablet replica
I20260619 14:57:41.794610 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 17394
I20260619 14:57:41.795346 18021 tablet_replica.cc:330] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: stopping tablet replica
I20260619 14:57:41.795470 18021 raft_consensus.cc:2229] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:41.795576 18021 raft_consensus.cc:2258] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:41.800547 18022 raft_consensus.cc:2229] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:41.800773 18022 raft_consensus.cc:2258] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:41.800980 18021 ts_tablet_manager.cc:1905] T 395959c3018e4c05801479efa7efc202 P 6eeb517267474aafaba6681953067269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:41.801311 18022 ts_tablet_manager.cc:1905] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:41.802810 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 17627
W20260619 14:57:41.802860 17996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:41.802445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:46755 (local address 127.0.0.1:34586)
0619 14:57:41.802709 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.802764 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:46755: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":177}
I20260619 14:57:41.803676 18022 ts_tablet_manager.cc:1918] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 14:57:41.803673 17232 catalog_manager.cc:4584] TS 65529b9919b04d4b923c9b3c52f0855d (127.10.71.129:46755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 395959c3018e4c05801479efa7efc202: Network error: Client connection negotiation failed: client connection to 127.10.71.129:46755: connect: Connection reset by peer (error 104)
I20260619 14:57:41.803745 18022 log.cc:1196] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/395959c3018e4c05801479efa7efc202
I20260619 14:57:41.804069 18022 ts_tablet_manager.cc:1939] T 395959c3018e4c05801479efa7efc202 P 8e0b534270fb49cd8d5016202e853065: Deleting consensus metadata
I20260619 14:57:41.804706 17232 catalog_manager.cc:4841] TS 8e0b534270fb49cd8d5016202e853065 (127.10.71.131:42443): tablet 395959c3018e4c05801479efa7efc202 (table test-table [id=f28e44b750fa4062a53954370121d8d4]) successfully deleted
I20260619 14:57:41.808741 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 17763
I20260619 14:57:41.809881 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:41.809598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40103 (local address 127.0.0.1:60578)
0619 14:57:41.809707 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:41.809765 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.130:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:57:41.814190 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 17114
I20260619 14:57:41.819641 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 17301
2026-06-19T14:57:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6791 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:41 dist-test-slave-3qdh krb5kdc[18025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:41 dist-test-slave-3qdh krb5kdc[18025](info): set up 2 sockets
Jun 19 14:57:41 dist-test-slave-3qdh krb5kdc[18025](info): commencing operation
krb5kdc: starting...
W20260619 14:57:42.764873 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:42 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab.
Entry for principal oryx/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab.
I20260619 14:57:42.879818 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37049
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab
--principal=oryx/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:42.978178 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:42.978441 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:42.978508 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:42.982216 18038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:42.982283 18038 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:57:42.982309 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:42.982363 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:42.982412 18038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:42.982445 18038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:42.987268 18038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:37049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.190.keytab
--principal=oryx/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:42.988193 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:42.989143 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:57:42.996816 18038 server_base.cc:1034] running on GCE node
W20260619 14:57:42.996836 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:42.996833 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:42.996807 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:42.997434 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:42.998342 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:42.999531 18038 hybrid_clock.cc:648] HybridClock initialized: now 1781881062999509 us; error 35 us; skew 500 ppm
Jun 19 14:57:42 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881062, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:43.001839 18038 init.cc:377] Logged in from keytab as oryx/127.10.71.190@KRBTEST.COM (short username oryx)
I20260619 14:57:43.002545 18038 webserver.cc:460] Webserver started at http://127.10.71.190:43257/ using document root <none> and password file <none>
I20260619 14:57:43.003018 18038 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:43.003105 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:43.003297 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:43.005165 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "f633206119104ec3898666356168e987"
format_stamp: "Formatted at 2026-06-19 14:57:42 on dist-test-slave-3qdh"
server_key: "b9f378f8ab4534e02d7df2953d57f430"
server_key_iv: "408d7abb4d6ec4b54e909c505922c8fe"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.005654 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "f633206119104ec3898666356168e987"
format_stamp: "Formatted at 2026-06-19 14:57:42 on dist-test-slave-3qdh"
server_key: "b9f378f8ab4534e02d7df2953d57f430"
server_key_iv: "408d7abb4d6ec4b54e909c505922c8fe"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.008764 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:43.010756 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.011845 18038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:43.011966 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "f633206119104ec3898666356168e987"
format_stamp: "Formatted at 2026-06-19 14:57:42 on dist-test-slave-3qdh"
server_key: "b9f378f8ab4534e02d7df2953d57f430"
server_key_iv: "408d7abb4d6ec4b54e909c505922c8fe"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.012059 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:43.033576 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:43.036182 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:43.036394 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:43.043385 18038 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:37049
I20260619 14:57:43.043416 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:37049 every 8 connection(s)
I20260619 14:57:43.044224 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:43.044899 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18038
I20260619 14:57:43.045030 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:43.045333 10526 external_mini_cluster.cc:1428] Setting key 93d952d2816f1eca0757d8bf177dde1a
I20260619 14:57:43.047142 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:43.053030 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Bootstrap starting.
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:43.055222 18116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:43.056154 18116 log.cc:824] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:43.058000 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: No bootstrap required, opened a new log
I20260619 14:57:43.058050 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.046536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.10.71.190:37049)
0619 14:57:43.046889 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.046902 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.046930 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:57:43.047544 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.047560 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.047855 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.048093 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.048977 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.049906 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.050882 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.051265 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.051362 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.053867 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.053871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.053882 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.053906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.055512 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.055893 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.055901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.055904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.055993 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.056258 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.056262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.056264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.056750 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.056942 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.057179 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.057437 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260619 14:57:43.060596 18116 raft_consensus.cc:348] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f633206119104ec3898666356168e987" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37049 } }
I20260619 14:57:43.060858 18116 raft_consensus.cc:374] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:43.060927 18116 raft_consensus.cc:729] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f633206119104ec3898666356168e987, State: Initialized, Role: FOLLOWER
I20260619 14:57:43.061300 18116 consensus_queue.cc:260] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f633206119104ec3898666356168e987" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37049 } }
I20260619 14:57:43.061431 18116 raft_consensus.cc:388] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:43.061507 18116 raft_consensus.cc:482] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:43.061610 18116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:43.062346 18116 raft_consensus.cc:504] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f633206119104ec3898666356168e987" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37049 } }
I20260619 14:57:43.062654 18116 leader_election.cc:302] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f633206119104ec3898666356168e987; no voters:
I20260619 14:57:43.062913 18116 leader_election.cc:288] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:43.063033 18121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:43.063287 18121 raft_consensus.cc:686] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [term 1 LEADER]: Becoming Leader. State: Replica: f633206119104ec3898666356168e987, State: Running, Role: LEADER
I20260619 14:57:43.063606 18121 consensus_queue.cc:237] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f633206119104ec3898666356168e987" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37049 } }
I20260619 14:57:43.064077 18116 sys_catalog.cc:564] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:43.065235 18123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f633206119104ec3898666356168e987. Latest consensus state: current_term: 1 leader_uuid: "f633206119104ec3898666356168e987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f633206119104ec3898666356168e987" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37049 } } }
I20260619 14:57:43.065353 18123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:43.066250 18130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:43.065690 18122 sys_catalog.cc:455] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f633206119104ec3898666356168e987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f633206119104ec3898666356168e987" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37049 } } }
I20260619 14:57:43.066673 18122 sys_catalog.cc:458] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:43.068626 18130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:43.072988 18130 catalog_manager.cc:1347] Generated new cluster ID: be8b2fa769b949508c9c7d4bb8cf5224
I20260619 14:57:43.073051 18130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:43.086376 18130 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:43.087155 18130 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:43.090718 18130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Generated new TSK 0
I20260619 14:57:43.091220 18130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab.
Entry for principal oryx/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab.
I20260619 14:57:43.117038 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37049
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:43.213577 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:43.213771 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:43.213809 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:43.217540 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:43.217581 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:43.217619 18142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:43.217651 18142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:43.221776 18142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:37049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:43.222476 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:43.223177 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:43.231477 18147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:43.231539 18148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:43.231469 18150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:43.231637 18142 server_base.cc:1034] running on GCE node
I20260619 14:57:43.232036 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:43.232574 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:43.233716 18142 hybrid_clock.cc:648] HybridClock initialized: now 1781881063233701 us; error 25 us; skew 500 ppm
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:43.236099 18142 init.cc:377] Logged in from keytab as oryx/127.10.71.129@KRBTEST.COM (short username oryx)
I20260619 14:57:43.236797 18142 webserver.cc:460] Webserver started at http://127.10.71.129:41807/ using document root <none> and password file <none>
I20260619 14:57:43.237319 18142 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:43.237396 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:43.237586 18142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:43.239180 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "754bbe55cd414778b19ef8b49f63b024"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "86d7e1ca945afc6749d5b093626c889a"
server_key_iv: "193e5c963ee3e2f113c382bc08941fd1"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.239634 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "754bbe55cd414778b19ef8b49f63b024"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "86d7e1ca945afc6749d5b093626c889a"
server_key_iv: "193e5c963ee3e2f113c382bc08941fd1"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.242579 18142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:57:43.244689 18157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.245733 18142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 14:57:43.245862 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "754bbe55cd414778b19ef8b49f63b024"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "86d7e1ca945afc6749d5b093626c889a"
server_key_iv: "193e5c963ee3e2f113c382bc08941fd1"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.245955 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:43.259698 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:43.262388 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:43.262567 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:43.263664 18142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:43.263728 18142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.263785 18142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:43.263823 18142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:43.273245 18142 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:33661
I20260619 14:57:43.273329 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:33661 every 8 connection(s)
I20260619 14:57:43.273993 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:43.275944 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.264503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:33485 (local address 127.10.71.190:37049)
0619 14:57:43.264625 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.264629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.265544 (+ 915us) server_negotiation.cc:407] Connection header received
0619 14:57:43.266328 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.266331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.266384 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.266466 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.267389 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.268125 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.269751 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.270083 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.270203 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.272695 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.272698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.272700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.272720 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.274225 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.274761 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.274763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.274765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.274836 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.275139 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.275141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.275142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.275274 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.275338 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.275752 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.275846 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 14:57:43.276633 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.264913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.10.71.129:33485)
0619 14:57:43.265364 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.265415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.266153 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.266612 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.266619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.266920 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:43.267267 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.267277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.268265 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.268269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.269610 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.269635 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.270267 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.270275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.270478 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.270947 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:43.270969 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:43.272559 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:43.274333 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.274338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.274349 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.274666 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.274918 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.274921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.274922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.275036 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.275348 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:43.275352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:43.275539 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.275694 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.275885 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260619 14:57:43.277890 18273 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37049
I20260619 14:57:43.278023 18273 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:43.278476 18273 heartbeater.cc:511] Master 127.10.71.190:37049 requested a full tablet report, sending...
I20260619 14:57:43.279897 18070 ts_manager.cc:194] Registered new tserver with Master: 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661)
I20260619 14:57:43.281132 18070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:33485
I20260619 14:57:43.281587 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18142
I20260619 14:57:43.281710 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:43.282089 10526 external_mini_cluster.cc:1428] Setting key acfdcbe0be70d64d63ff9ab94846a2b0
I20260619 14:57:43.286525 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.279859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:56365 (local address 127.10.71.190:37049)
0619 14:57:43.279948 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.279952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.280104 (+ 152us) server_negotiation.cc:407] Connection header received
0619 14:57:43.280172 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.280174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.280205 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.280275 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.280634 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.281231 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.282264 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.282528 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.282614 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.283535 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.283540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.283545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.283569 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.285138 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.285579 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.285581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.285582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.285647 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.285914 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.285915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.285916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.286040 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.286102 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.286356 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.286426 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:57:43.286727 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.279821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.10.71.129:56365)
0619 14:57:43.280028 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.280039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.280121 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.280280 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.280281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.280461 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:43.280534 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.280538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.281326 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.281328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.282155 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.282166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.282619 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.282623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.282703 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.283122 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:43.283134 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:43.283420 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:43.285246 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.285251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.285254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.285487 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.285722 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.285728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.285731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.285827 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.286123 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:43.286128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:43.286222 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.286329 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.286462 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab.
Entry for principal oryx/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab.
I20260619 14:57:43.305253 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37049
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:43.399677 18282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:43.399890 18282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:43.399982 18282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:43.403823 18282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:43.403867 18282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:43.403932 18282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:43.404036 18282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:43.408856 18282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:37049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:43.409646 18282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:43.410404 18282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:43.418898 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:43.418905 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:43.419059 18282 server_base.cc:1034] running on GCE node
W20260619 14:57:43.418943 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:43.419453 18282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:43.419941 18282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:43.421130 18282 hybrid_clock.cc:648] HybridClock initialized: now 1781881063421125 us; error 28 us; skew 500 ppm
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:43.423259 18282 init.cc:377] Logged in from keytab as oryx/127.10.71.130@KRBTEST.COM (short username oryx)
I20260619 14:57:43.423946 18282 webserver.cc:460] Webserver started at http://127.10.71.130:39255/ using document root <none> and password file <none>
I20260619 14:57:43.424501 18282 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:43.424593 18282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:43.424798 18282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:43.426438 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "0e1176ed6f184e8592d86ad185919f14"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "d745c914d08cd8c2078f2fcc64b9c5d2"
server_key_iv: "9b06d9535d4e33f52b7a735c676b2c46"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.426926 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "0e1176ed6f184e8592d86ad185919f14"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "d745c914d08cd8c2078f2fcc64b9c5d2"
server_key_iv: "9b06d9535d4e33f52b7a735c676b2c46"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.430145 18282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:43.431977 18297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.432960 18282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:43.433070 18282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "0e1176ed6f184e8592d86ad185919f14"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "d745c914d08cd8c2078f2fcc64b9c5d2"
server_key_iv: "9b06d9535d4e33f52b7a735c676b2c46"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.433157 18282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:43.442536 18282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:43.445077 18282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:43.445271 18282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:43.446441 18282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:43.446513 18282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.446566 18282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:43.446616 18282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:43.456072 18282 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:36677
I20260619 14:57:43.456377 18412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:36677 every 8 connection(s)
I20260619 14:57:43.456895 18282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:57:43.458686 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.447080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:49535 (local address 127.10.71.190:37049)
0619 14:57:43.447225 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.447230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.447825 (+ 595us) server_negotiation.cc:407] Connection header received
0619 14:57:43.448703 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.448706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.448758 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.448833 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.450061 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.450667 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.451819 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.452112 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.452219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.455348 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.455352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.455354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.455379 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.456798 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.457352 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.457355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.457357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.457436 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.457783 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.457788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.457789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.457942 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.458033 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.458440 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.458538 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:57:43.459220 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.447342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.10.71.130:49535)
0619 14:57:43.447696 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.447734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.448523 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.449075 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.449148 (+ 73us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.449446 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:43.449928 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.449939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.450781 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.450785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.451708 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.451718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.452257 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.452261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.452441 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.452929 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:43.452946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:43.455157 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:43.456900 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.456906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.456920 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.457230 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.457530 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.457534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.457537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.457680 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.458042 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:43.458049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:43.458241 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.458389 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.458627 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
I20260619 14:57:43.459792 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18282
I20260619 14:57:43.459908 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:43.460201 10526 external_mini_cluster.cc:1428] Setting key fd6fe33efaa6f2e82da505e64e93eff8
I20260619 14:57:43.461933 18413 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37049
I20260619 14:57:43.462172 18413 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:43.462517 18413 heartbeater.cc:511] Master 127.10.71.190:37049 requested a full tablet report, sending...
I20260619 14:57:43.463342 18069 ts_manager.cc:194] Registered new tserver with Master: 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677)
I20260619 14:57:43.464166 18069 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:49535
I20260619 14:57:43.469732 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.462100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.10.71.130:39741)
0619 14:57:43.463046 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.463067 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.463156 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.463368 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.463370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.463534 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:43.463643 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.463649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.464729 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.464732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.465385 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.465393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.465885 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.465887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.465943 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.466292 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:43.466302 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:43.466559 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:43.468387 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.468390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.468391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.468560 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.468849 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.468851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.468852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.468918 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.469381 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:43.469383 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:43.469446 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.469528 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.469606 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":77,"threads_started":1}
I20260619 14:57:43.470031 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.462096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:39741 (local address 127.10.71.190:37049)
0619 14:57:43.462219 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.462225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.463126 (+ 901us) server_negotiation.cc:407] Connection header received
0619 14:57:43.463240 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.463243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.463277 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.463355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.463741 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.464593 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.465492 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.465770 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.465874 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.466688 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.466694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.466700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.466727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.468242 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.468657 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.468662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.468666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.468754 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.469002 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.469008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.469012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.469284 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.469383 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.469614 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.469731 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab.
Entry for principal oryx/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab.
I20260619 14:57:43.486475 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37049
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:43.581676 18422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:43.581852 18422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:43.581887 18422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:43.585649 18422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:43.585691 18422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:43.585747 18422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:57:43.585800 18422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:43.590289 18422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.71.190:37049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:43.591033 18422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:43.591815 18422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:43.599821 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:43.599877 18422 server_base.cc:1034] running on GCE node
W20260619 14:57:43.599821 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:43.599843 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:43.600399 18422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:43.600907 18422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:43.602066 18422 hybrid_clock.cc:648] HybridClock initialized: now 1781881063602032 us; error 36 us; skew 500 ppm
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:43.604264 18422 init.cc:377] Logged in from keytab as oryx/127.10.71.131@KRBTEST.COM (short username oryx)
I20260619 14:57:43.605062 18422 webserver.cc:460] Webserver started at http://127.10.71.131:32807/ using document root <none> and password file <none>
I20260619 14:57:43.605551 18422 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:43.605630 18422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:43.605818 18422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:43.607402 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "ac049f724bf74d0a8b336a86a34d8d98"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "50a3bafde4dcb04e9a1644436824e42e"
server_key_iv: "ff820dc35a8e668c633c7b8a589dd0c8"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.607860 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "ac049f724bf74d0a8b336a86a34d8d98"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "50a3bafde4dcb04e9a1644436824e42e"
server_key_iv: "ff820dc35a8e668c633c7b8a589dd0c8"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.610742 18422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:43.612764 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.613893 18422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:43.614069 18422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "ac049f724bf74d0a8b336a86a34d8d98"
format_stamp: "Formatted at 2026-06-19 14:57:43 on dist-test-slave-3qdh"
server_key: "50a3bafde4dcb04e9a1644436824e42e"
server_key_iv: "ff820dc35a8e668c633c7b8a589dd0c8"
server_key_version: "encryptionkey@0"
I20260619 14:57:43.614173 18422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:43.636368 18422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:43.638720 18422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:43.638883 18422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:43.640122 18422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:43.640184 18422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:43.640242 18422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:43.640273 18422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:43.650167 18422 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:37977
I20260619 14:57:43.650200 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:37977 every 8 connection(s)
I20260619 14:57:43.650898 18422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:43.651126 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18422
I20260619 14:57:43.651254 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:43.651501 10526 external_mini_cluster.cc:1428] Setting key 7a8990d7cef69a64b03c6e69420ece04
I20260619 14:57:43.652947 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.640651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:40535 (local address 127.10.71.190:37049)
0619 14:57:43.640773 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.640777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.641589 (+ 812us) server_negotiation.cc:407] Connection header received
0619 14:57:43.642638 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.642641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.642690 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.642797 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.644174 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.644927 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.645990 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.646221 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.646337 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.649469 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.649472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.649474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.649495 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.650893 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.651499 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.651504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.651506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.651603 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.651975 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.651978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.651979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.652140 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.652243 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.652695 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.652799 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:57:43.653707 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.641021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.10.71.131:40535)
0619 14:57:43.641430 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.641479 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.642424 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.642936 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.642947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.643386 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:43.643965 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.644003 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.645048 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.645054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.645872 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.645882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.646380 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.646383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.646519 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.647030 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:43.647047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:43.649306 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:43.651035 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.651040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.651051 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.651383 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.651698 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.651702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.651704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.651870 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.652249 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:43.652256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:43.652485 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.652643 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.652892 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260619 14:57:43.654676 18553 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37049
I20260619 14:57:43.654798 18553 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:43.655248 18553 heartbeater.cc:511] Master 127.10.71.190:37049 requested a full tablet report, sending...
I20260619 14:57:43.656098 18070 ts_manager.cc:194] Registered new tserver with Master: ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977)
I20260619 14:57:43.656819 18070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:40535
I20260619 14:57:43.662379 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.656366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:51157 (local address 127.10.71.190:37049)
0619 14:57:43.656477 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.656481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.656601 (+ 120us) server_negotiation.cc:407] Connection header received
0619 14:57:43.656698 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.656700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.656729 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.656784 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.657186 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.657606 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.658421 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.658634 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.658738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.659650 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.659657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.659659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.659677 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.660837 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.661241 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.661246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.661247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.661374 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.661640 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.661644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.661646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.661789 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.661886 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.662130 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.662244 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:57:43.662386 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.656289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.10.71.131:51157)
0619 14:57:43.656542 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.656554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.656633 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.656819 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.656821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.656989 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:43.657107 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.657113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.657725 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.657728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.658306 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.658313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.658742 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.658746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.658833 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.659243 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:43.659253 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:43.659504 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:43.660944 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.660948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.660950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.661145 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.661474 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:43.661477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:43.661478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:43.661556 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:43.661885 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:43.661888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:43.661964 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.662079 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.662213 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260619 14:57:43.665187 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:57:43.686319 18119 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:43.680811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.10.71.190:37049)
0619 14:57:43.680917 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.680920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.680931 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:43.680962 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.680964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.680995 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.681059 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.681670 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.682109 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.682960 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.683170 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.683252 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.686165 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59794: BlockingRecv error: recv got EOF from 127.0.0.1:59794 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 14:57:43 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881063, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:43.693969 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.687571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.10.71.190:37049)
0619 14:57:43.687709 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.687712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.687763 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:57:43.687867 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.687869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.687906 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.687965 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:43.688503 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.688919 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.689549 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.689732 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.689795 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.691535 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:43.691537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:43.691539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:43.691557 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:43.692769 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.693120 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.693122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.693123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.693184 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:43.693366 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:43.693368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:43.693369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:43.693493 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:43.693576 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.693724 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.693795 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:57:43.696604 18070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:43.699349 18070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:43.712252 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.707279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33661 (local address 127.0.0.1:59720)
0619 14:57:43.707886 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.707895 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.707966 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.708596 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.708598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.708615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:43.708696 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.708702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.709599 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.709602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.711044 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.711053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.711749 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.711852 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.711910 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.712028 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.712138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":205,"thread_start_us":89,"threads_started":1}
I20260619 14:57:43.712265 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.707821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.10.71.129:33661)
0619 14:57:43.708132 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.708143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.708163 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:57:43.708232 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.708238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.708426 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.708599 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:43.708794 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.709490 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711153 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.711636 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.711893 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.712056 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.712138 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260619 14:57:43.712441 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.706698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:36677 (local address 127.0.0.1:34358)
0619 14:57:43.707524 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.707569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.707701 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.708242 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.708247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.708267 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:43.708409 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.708415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.709643 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.709645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.710983 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.710993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.712043 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.712046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.712135 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.712150 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.712237 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.712304 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":73,"thread_start_us":90,"threads_started":1}
I20260619 14:57:43.712502 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.707090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.10.71.130:36677)
0619 14:57:43.707354 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.707364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.707650 (+ 286us) server_negotiation.cc:407] Connection header received
0619 14:57:43.707876 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.707885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.708096 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.708253 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:43.708515 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.709555 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711107 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.711756 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.711985 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.712276 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.712378 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
I20260619 14:57:43.712934 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.706648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37977 (local address 127.0.0.1:54602)
0619 14:57:43.707524 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.707576 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.707678 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.708727 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.708730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.708745 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:43.708850 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.708855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.710033 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.710036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.710869 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.710878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.711848 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.712646 (+ 795us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.712675 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.712744 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.712810 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":673,"thread_start_us":61,"threads_started":1}
I20260619 14:57:43.713289 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.706754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.10.71.131:37977)
0619 14:57:43.708188 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.708198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.708214 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:57:43.708269 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.708273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.708427 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.708600 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:43.708965 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.709919 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.710989 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.711716 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.711892 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.712019 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.713073 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.713151 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":97,"threads_started":1}
I20260619 14:57:43.715039 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 797997ab280f4ab7bc906eac80aca771 (DEFAULT_TABLE table=test-table [id=96a0f5808e494281a79c4cdb18fc8188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:43.715042 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 797997ab280f4ab7bc906eac80aca771 (DEFAULT_TABLE table=test-table [id=96a0f5808e494281a79c4cdb18fc8188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:43.715920 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797997ab280f4ab7bc906eac80aca771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:43.715920 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797997ab280f4ab7bc906eac80aca771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:43.722014 18581 tablet_bootstrap.cc:492] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Bootstrap starting.
I20260619 14:57:43.722646 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 797997ab280f4ab7bc906eac80aca771 (DEFAULT_TABLE table=test-table [id=96a0f5808e494281a79c4cdb18fc8188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:43.723342 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797997ab280f4ab7bc906eac80aca771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:43.723766 18581 tablet_bootstrap.cc:654] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:43.724546 18581 log.cc:824] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:43.724872 18582 tablet_bootstrap.cc:492] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Bootstrap starting.
I20260619 14:57:43.726480 18581 tablet_bootstrap.cc:492] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: No bootstrap required, opened a new log
I20260619 14:57:43.726650 18581 ts_tablet_manager.cc:1397] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:43.726809 18582 tablet_bootstrap.cc:654] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:43.727531 18582 log.cc:824] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:43.729204 18585 tablet_bootstrap.cc:492] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Bootstrap starting.
I20260619 14:57:43.729269 18582 tablet_bootstrap.cc:492] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: No bootstrap required, opened a new log
I20260619 14:57:43.729451 18582 ts_tablet_manager.cc:1397] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:43.730145 18581 raft_consensus.cc:348] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.730382 18581 raft_consensus.cc:374] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:43.730446 18581 raft_consensus.cc:729] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1176ed6f184e8592d86ad185919f14, State: Initialized, Role: FOLLOWER
I20260619 14:57:43.730888 18581 consensus_queue.cc:260] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.731508 18413 heartbeater.cc:503] Master 127.10.71.190:37049 was elected leader, sending a full tablet report...
I20260619 14:57:43.731630 18585 tablet_bootstrap.cc:654] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:43.731818 18581 ts_tablet_manager.cc:1428] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:57:43.732352 18585 log.cc:824] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:43.732956 18582 raft_consensus.cc:348] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.733173 18582 raft_consensus.cc:374] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:43.733235 18582 raft_consensus.cc:729] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754bbe55cd414778b19ef8b49f63b024, State: Initialized, Role: FOLLOWER
I20260619 14:57:43.733589 18582 consensus_queue.cc:260] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.734150 18585 tablet_bootstrap.cc:492] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: No bootstrap required, opened a new log
I20260619 14:57:43.734316 18585 ts_tablet_manager.cc:1397] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:57:43.734324 18273 heartbeater.cc:503] Master 127.10.71.190:37049 was elected leader, sending a full tablet report...
I20260619 14:57:43.734546 18582 ts_tablet_manager.cc:1428] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260619 14:57:43.738148 18585 raft_consensus.cc:348] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.738306 18585 raft_consensus.cc:374] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:43.738339 18585 raft_consensus.cc:729] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Initialized, Role: FOLLOWER
I20260619 14:57:43.738590 18585 consensus_queue.cc:260] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.739049 18553 heartbeater.cc:503] Master 127.10.71.190:37049 was elected leader, sending a full tablet report...
I20260619 14:57:43.739221 18585 ts_tablet_manager.cc:1428] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 14:57:43.775020 18274 tablet.cc:2369] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:57:43.902071 18554 tablet.cc:2369] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:43.951340 18589 raft_consensus.cc:482] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:43.951553 18589 raft_consensus.cc:504] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.952798 18589 leader_election.cc:288] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677), 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661)
I20260619 14:57:43.956612 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.953002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:36677 (local address 127.10.71.131:39703)
0619 14:57:43.953185 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.953204 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.953309 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.953607 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.953611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.953628 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:43.953753 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.953759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.954625 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.954632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.955437 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.955447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.956149 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.956153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.956271 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.956290 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.956414 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.956486 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 14:57:43.956612 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.953051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:39703 (local address 127.10.71.130:36677)
0619 14:57:43.953175 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.953179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.953276 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:57:43.953458 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.953461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.953505 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.953596 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:43.953869 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.954507 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.955607 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.956008 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.956149 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.956207 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.956427 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.956494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:43.957140 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.953234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:55715 (local address 127.10.71.129:33661)
0619 14:57:43.953378 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.953383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.953543 (+ 160us) server_negotiation.cc:407] Connection header received
0619 14:57:43.953705 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.953710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.953753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.953827 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:43.954532 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.955124 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.956152 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.956598 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.956718 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.956759 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:43.956977 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:57:43.957037 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:57:43.957154 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.953064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33661 (local address 127.10.71.131:55715)
0619 14:57:43.953459 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:43.953469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:43.953598 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:43.953846 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:43.953849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:43.953863 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:43.954409 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.954417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.955237 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.955242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.956029 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:43.956037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.956723 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:43.956726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:43.956817 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:43.956842 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:57:43.956959 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:57:43.957023 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
I20260619 14:57:43.957319 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797997ab280f4ab7bc906eac80aca771" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14" is_pre_election: true
I20260619 14:57:43.957634 18367 raft_consensus.cc:2454] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 0.
W20260619 14:57:43.957928 18414 tablet.cc:2369] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:57:43.957887 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797997ab280f4ab7bc906eac80aca771" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024" is_pre_election: true
I20260619 14:57:43.958140 18438 leader_election.cc:302] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:43.958180 18227 raft_consensus.cc:2454] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 0.
I20260619 14:57:43.958405 18589 raft_consensus.cc:2781] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:43.958510 18589 raft_consensus.cc:482] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:43.958576 18589 raft_consensus.cc:3037] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:43.959523 18589 raft_consensus.cc:504] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.959872 18589 leader_election.cc:288] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 election: Requested vote from peers 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677), 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661)
I20260619 14:57:43.960239 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797997ab280f4ab7bc906eac80aca771" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14"
I20260619 14:57:43.960182 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797997ab280f4ab7bc906eac80aca771" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024"
I20260619 14:57:43.960421 18367 raft_consensus.cc:3037] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:43.960453 18227 raft_consensus.cc:3037] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:43.961505 18367 raft_consensus.cc:2454] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 1.
I20260619 14:57:43.961750 18438 leader_election.cc:302] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:43.961776 18227 raft_consensus.cc:2454] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 1.
I20260619 14:57:43.961922 18589 raft_consensus.cc:2781] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:43.962149 18589 raft_consensus.cc:686] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 LEADER]: Becoming Leader. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Running, Role: LEADER
I20260619 14:57:43.962442 18589 consensus_queue.cc:237] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } }
I20260619 14:57:43.965662 18070 catalog_manager.cc:5495] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:44.001895 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:43.998409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.10.71.131:37977)
0619 14:57:43.998546 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:43.998551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:43.998565 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:43.998717 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:43.998720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:43.998767 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:43.998869 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:43.999080 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:43.999594 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.000503 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.000727 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.000832 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.001101 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.001493 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.001642 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.001697 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.001762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:57:44.007385 18227 raft_consensus.cc:1261] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Refusing update from remote peer ac049f724bf74d0a8b336a86a34d8d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.007387 18367 raft_consensus.cc:1261] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Refusing update from remote peer ac049f724bf74d0a8b336a86a34d8d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.007987 18589 consensus_queue.cc:1035] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.008186 18591 consensus_queue.cc:1035] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.016388 18595 mvcc.cc:205] Tried to move back new op lower bound from 7298584838167453696 to 7298584837992472576. Current Snapshot: MvccSnapshot[applied={T|T < 7298584838167453696}]
I20260619 14:57:44.019119 18596 mvcc.cc:205] Tried to move back new op lower bound from 7298584838167453696 to 7298584837992472576. Current Snapshot: MvccSnapshot[applied={T|T < 7298584838167453696 or (T in {7298584838167453696})}]
I20260619 14:57:44.021785 18070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.021934 18070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.024143 18070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Sending DeleteTablet for 3 replicas of tablet 797997ab280f4ab7bc906eac80aca771
I20260619 14:57:44.024847 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet 797997ab280f4ab7bc906eac80aca771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:59720
I20260619 14:57:44.024863 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet 797997ab280f4ab7bc906eac80aca771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34358
I20260619 14:57:44.024866 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 797997ab280f4ab7bc906eac80aca771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:54602
I20260619 14:57:44.025332 18604 tablet_replica.cc:330] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: stopping tablet replica
I20260619 14:57:44.025624 18604 raft_consensus.cc:2229] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.025657 18070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:57:44.025879 18604 raft_consensus.cc:2258] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:57:44.026041 18070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:44.026171 18605 tablet_replica.cc:330] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: stopping tablet replica
I20260619 14:57:44.026417 18605 raft_consensus.cc:2229] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.026678 18605 raft_consensus.cc:2258] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.026952 18606 tablet_replica.cc:330] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: stopping tablet replica
I20260619 14:57:44.027182 18606 raft_consensus.cc:2229] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:44.027470 18606 raft_consensus.cc:2258] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.027957 18605 ts_tablet_manager.cc:1905] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.028434 18604 ts_tablet_manager.cc:1905] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.028721 18606 ts_tablet_manager.cc:1905] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.030483 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 316f621b52044ea1981d90627e457a9e (DEFAULT_TABLE table=test-table [id=e22f5077d7ea4b56861229294eba28b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.030604 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 316f621b52044ea1981d90627e457a9e (DEFAULT_TABLE table=test-table [id=e22f5077d7ea4b56861229294eba28b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.030714 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316f621b52044ea1981d90627e457a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.030723 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 316f621b52044ea1981d90627e457a9e (DEFAULT_TABLE table=test-table [id=e22f5077d7ea4b56861229294eba28b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.030821 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316f621b52044ea1981d90627e457a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.030946 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316f621b52044ea1981d90627e457a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.032828 18582 tablet_bootstrap.cc:492] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Bootstrap starting.
I20260619 14:57:44.032848 18581 tablet_bootstrap.cc:492] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Bootstrap starting.
I20260619 14:57:44.033005 18585 tablet_bootstrap.cc:492] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Bootstrap starting.
I20260619 14:57:44.033596 18581 tablet_bootstrap.cc:654] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.033703 18582 tablet_bootstrap.cc:654] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.033818 18585 tablet_bootstrap.cc:654] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.035050 18585 tablet_bootstrap.cc:492] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: No bootstrap required, opened a new log
I20260619 14:57:44.035130 18585 ts_tablet_manager.cc:1397] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:44.035256 18606 ts_tablet_manager.cc:1918] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:44.035334 18606 log.cc:1196] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/797997ab280f4ab7bc906eac80aca771
I20260619 14:57:44.035650 18606 ts_tablet_manager.cc:1939] T 797997ab280f4ab7bc906eac80aca771 P ac049f724bf74d0a8b336a86a34d8d98: Deleting consensus metadata
I20260619 14:57:44.035611 18585 raft_consensus.cc:348] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.035709 18585 raft_consensus.cc:374] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.035739 18585 raft_consensus.cc:729] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.035842 18585 consensus_queue.cc:260] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.036067 18585 ts_tablet_manager.cc:1428] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:44.036350 18604 ts_tablet_manager.cc:1918] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:44.036432 18604 log.cc:1196] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/797997ab280f4ab7bc906eac80aca771
I20260619 14:57:44.036782 18604 ts_tablet_manager.cc:1939] T 797997ab280f4ab7bc906eac80aca771 P 0e1176ed6f184e8592d86ad185919f14: Deleting consensus metadata
I20260619 14:57:44.036839 18056 catalog_manager.cc:4841] TS ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977): tablet 797997ab280f4ab7bc906eac80aca771 (table test-table [id=96a0f5808e494281a79c4cdb18fc8188]) successfully deleted
I20260619 14:57:44.038026 18054 catalog_manager.cc:4841] TS 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677): tablet 797997ab280f4ab7bc906eac80aca771 (table test-table [id=96a0f5808e494281a79c4cdb18fc8188]) successfully deleted
I20260619 14:57:44.038357 18605 ts_tablet_manager.cc:1918] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:44.038404 18581 tablet_bootstrap.cc:492] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: No bootstrap required, opened a new log
I20260619 14:57:44.038424 18605 log.cc:1196] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/797997ab280f4ab7bc906eac80aca771
I20260619 14:57:44.038482 18581 ts_tablet_manager.cc:1397] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 14:57:44.038839 18581 raft_consensus.cc:348] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.038952 18581 raft_consensus.cc:374] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.038981 18605 ts_tablet_manager.cc:1939] T 797997ab280f4ab7bc906eac80aca771 P 754bbe55cd414778b19ef8b49f63b024: Deleting consensus metadata
I20260619 14:57:44.039009 18581 raft_consensus.cc:729] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1176ed6f184e8592d86ad185919f14, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.039152 18581 consensus_queue.cc:260] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.039422 18581 ts_tablet_manager.cc:1428] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:44.039996 18055 catalog_manager.cc:4841] TS 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661): tablet 797997ab280f4ab7bc906eac80aca771 (table test-table [id=96a0f5808e494281a79c4cdb18fc8188]) successfully deleted
I20260619 14:57:44.045408 18582 tablet_bootstrap.cc:492] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: No bootstrap required, opened a new log
I20260619 14:57:44.045490 18582 ts_tablet_manager.cc:1397] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20260619 14:57:44.045881 18582 raft_consensus.cc:348] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.045985 18582 raft_consensus.cc:374] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.046023 18582 raft_consensus.cc:729] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754bbe55cd414778b19ef8b49f63b024, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.046131 18582 consensus_queue.cc:260] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.046350 18582 ts_tablet_manager.cc:1428] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:44.159828 18601 raft_consensus.cc:482] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:44.159951 18601 raft_consensus.cc:504] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.160422 18601 leader_election.cc:288] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661), 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677)
I20260619 14:57:44.160815 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f621b52044ea1981d90627e457a9e" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024" is_pre_election: true
I20260619 14:57:44.160838 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f621b52044ea1981d90627e457a9e" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14" is_pre_election: true
I20260619 14:57:44.160959 18227 raft_consensus.cc:2454] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 0.
I20260619 14:57:44.160959 18367 raft_consensus.cc:2454] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 0.
I20260619 14:57:44.161243 18438 leader_election.cc:302] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:44.161401 18601 raft_consensus.cc:2781] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:44.161456 18601 raft_consensus.cc:482] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:44.161499 18601 raft_consensus.cc:3037] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.162384 18601 raft_consensus.cc:504] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.162725 18601 leader_election.cc:288] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 election: Requested vote from peers 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661), 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677)
I20260619 14:57:44.163028 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f621b52044ea1981d90627e457a9e" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024"
I20260619 14:57:44.163038 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f621b52044ea1981d90627e457a9e" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14"
I20260619 14:57:44.163113 18227 raft_consensus.cc:3037] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.163158 18367 raft_consensus.cc:3037] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.164003 18227 raft_consensus.cc:2454] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 1.
I20260619 14:57:44.164238 18367 raft_consensus.cc:2454] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 1.
I20260619 14:57:44.164354 18439 leader_election.cc:302] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754bbe55cd414778b19ef8b49f63b024, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:44.164585 18601 raft_consensus.cc:2781] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:44.164674 18601 raft_consensus.cc:686] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 1 LEADER]: Becoming Leader. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Running, Role: LEADER
I20260619 14:57:44.164800 18601 consensus_queue.cc:237] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.166452 18070 catalog_manager.cc:5495] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } health_report { overall_health: HEALTHY } } }
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881062, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.190@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:44.212831 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.205694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37049 (local address 127.0.0.1:59826)
0619 14:57:44.205989 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.206000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.206074 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.206275 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.206277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.206613 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:44.206693 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.206698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.207361 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.207362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.207881 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.207887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.208266 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.208269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.208374 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.208906 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:44.208917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:44.210198 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:44.211577 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:44.211583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:44.211588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:44.211786 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:44.212010 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:44.212012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:44.212013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:44.212086 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:44.212444 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:44.212448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:44.212532 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.212623 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.212717 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
I20260619 14:57:44.212986 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.205793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.10.71.190:37049)
0619 14:57:44.206035 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.206042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.206054 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:44.206148 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.206150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.206183 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.206244 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:44.206789 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.207270 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.207971 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.208167 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.208232 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.210306 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:44.210308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:44.210310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:44.210328 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:44.211485 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:44.211862 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:44.211864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:44.211865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:44.211931 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:44.212155 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:44.212157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:44.212158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:44.212337 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:44.212419 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.212696 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.212796 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260619 14:57:44.216369 18070 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:59826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:57:44.217038 18070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:44.221354 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 3565e448405243c3ae72a126f4b9c7a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d228c14f95d1409eabae30f51ce4eb5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:44.221383 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 3565e448405243c3ae72a126f4b9c7a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d228c14f95d1409eabae30f51ce4eb5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:44.221722 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565e448405243c3ae72a126f4b9c7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.221736 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565e448405243c3ae72a126f4b9c7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.223836 18582 tablet_bootstrap.cc:492] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: Bootstrap starting.
I20260619 14:57:44.223855 18581 tablet_bootstrap.cc:492] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: Bootstrap starting.
I20260619 14:57:44.224385 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 3565e448405243c3ae72a126f4b9c7a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d228c14f95d1409eabae30f51ce4eb5d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:57:44.224746 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565e448405243c3ae72a126f4b9c7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.224815 18581 tablet_bootstrap.cc:654] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.224850 18582 tablet_bootstrap.cc:654] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.225951 18582 tablet_bootstrap.cc:492] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: No bootstrap required, opened a new log
I20260619 14:57:44.226037 18582 ts_tablet_manager.cc:1397] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:44.226156 18581 tablet_bootstrap.cc:492] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: No bootstrap required, opened a new log
I20260619 14:57:44.226229 18581 ts_tablet_manager.cc:1397] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:44.226473 18582 raft_consensus.cc:348] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.226591 18582 raft_consensus.cc:374] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.226637 18582 raft_consensus.cc:729] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754bbe55cd414778b19ef8b49f63b024, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.226634 18581 raft_consensus.cc:348] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.226733 18581 raft_consensus.cc:374] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.226764 18581 raft_consensus.cc:729] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1176ed6f184e8592d86ad185919f14, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.226768 18582 consensus_queue.cc:260] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.226871 18581 consensus_queue.cc:260] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.227025 18582 ts_tablet_manager.cc:1428] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 14:57:44.227116 18581 ts_tablet_manager.cc:1428] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:44.227240 18586 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.227383 18586 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.227391 18585 tablet_bootstrap.cc:492] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: Bootstrap starting.
I20260619 14:57:44.228214 18585 tablet_bootstrap.cc:654] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.228230 18588 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.228384 18588 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.229269 18585 tablet_bootstrap.cc:492] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: No bootstrap required, opened a new log
I20260619 14:57:44.229350 18585 ts_tablet_manager.cc:1397] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:44.229678 18585 raft_consensus.cc:348] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.229780 18585 raft_consensus.cc:374] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.229810 18585 raft_consensus.cc:729] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.229926 18585 consensus_queue.cc:260] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.230180 18585 ts_tablet_manager.cc:1428] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:44.230129 18601 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.230263 18601 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.314738 18601 raft_consensus.cc:482] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:44.314877 18601 raft_consensus.cc:504] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.315392 18601 leader_election.cc:288] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661), 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677)
I20260619 14:57:44.315732 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565e448405243c3ae72a126f4b9c7a2" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024" is_pre_election: true
I20260619 14:57:44.315869 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565e448405243c3ae72a126f4b9c7a2" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14" is_pre_election: true
I20260619 14:57:44.315927 18227 raft_consensus.cc:2454] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 0.
I20260619 14:57:44.316039 18367 raft_consensus.cc:2454] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 0.
I20260619 14:57:44.316284 18439 leader_election.cc:302] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754bbe55cd414778b19ef8b49f63b024, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:44.316468 18601 raft_consensus.cc:2781] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:44.316529 18601 raft_consensus.cc:482] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:44.316594 18601 raft_consensus.cc:3037] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.317653 18601 raft_consensus.cc:504] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.318034 18601 leader_election.cc:288] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 election: Requested vote from peers 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661), 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677)
I20260619 14:57:44.318369 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565e448405243c3ae72a126f4b9c7a2" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024"
I20260619 14:57:44.318488 18227 raft_consensus.cc:3037] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.318534 18367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565e448405243c3ae72a126f4b9c7a2" candidate_uuid: "ac049f724bf74d0a8b336a86a34d8d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14"
I20260619 14:57:44.318670 18367 raft_consensus.cc:3037] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.319476 18227 raft_consensus.cc:2454] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 1.
I20260619 14:57:44.319592 18367 raft_consensus.cc:2454] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac049f724bf74d0a8b336a86a34d8d98 in term 1.
I20260619 14:57:44.319798 18439 leader_election.cc:302] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754bbe55cd414778b19ef8b49f63b024, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:44.320022 18601 raft_consensus.cc:2781] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:44.320134 18601 raft_consensus.cc:686] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 LEADER]: Becoming Leader. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Running, Role: LEADER
I20260619 14:57:44.320289 18601 consensus_queue.cc:237] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.320950 18591 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: TxnStatusTablet state changed. Reason: New leader ac049f724bf74d0a8b336a86a34d8d98. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.321058 18591 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:44.321296 18622 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:57:44.321482 18622 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:57:44.321905 18070 catalog_manager.cc:5495] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } health_report { overall_health: HEALTHY } } }
I20260619 14:57:44.352139 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.348939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.10.71.131:37977)
0619 14:57:44.349057 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.349061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.349103 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:57:44.349286 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.349289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.349337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.349409 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:44.349659 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.350345 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.351138 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.351438 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.351529 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.351758 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.351830 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.351892 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.351975 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.352022 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:57:44.352152 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.348825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37977 (local address 127.0.0.1:54612)
0619 14:57:44.348982 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.349000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.349178 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.349433 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.349434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.349441 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:44.349566 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.349571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.350458 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.350460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.351021 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.351029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.351539 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.351542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.351608 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.351647 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.351907 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.351916 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.351967 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.352033 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:57:44.354550 18367 raft_consensus.cc:1261] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Refusing update from remote peer ac049f724bf74d0a8b336a86a34d8d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.354557 18227 raft_consensus.cc:1261] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Refusing update from remote peer ac049f724bf74d0a8b336a86a34d8d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.354985 18601 consensus_queue.cc:1035] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.355115 18591 consensus_queue.cc:1035] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.356485 18586 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: TxnStatusTablet state changed. Reason: New leader ac049f724bf74d0a8b336a86a34d8d98. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.356597 18586 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.356565 18588 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: TxnStatusTablet state changed. Reason: New leader ac049f724bf74d0a8b336a86a34d8d98. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.356674 18588 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.358330 18591 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.358458 18591 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:44.360554 18586 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.360651 18586 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P 0e1176ed6f184e8592d86ad185919f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.361908 18594 mvcc.cc:205] Tried to move back new op lower bound from 7298584839592259584 to 7298584839457374208. Current Snapshot: MvccSnapshot[applied={T|T < 7298584839592259584 or (T in {7298584839592259584})}]
I20260619 14:57:44.363412 18601 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.363500 18601 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P ac049f724bf74d0a8b336a86a34d8d98: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:57:44.365217 18588 tablet_replica.cc:439] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac049f724bf74d0a8b336a86a34d8d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } } }
I20260619 14:57:44.365339 18588 tablet_replica.cc:442] T 3565e448405243c3ae72a126f4b9c7a2 P 754bbe55cd414778b19ef8b49f63b024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:57:44.372406 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.369132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:45337 (local address 127.10.71.131:37977)
0619 14:57:44.369230 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.369234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.369484 (+ 250us) server_negotiation.cc:407] Connection header received
0619 14:57:44.369584 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.369587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.369628 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.369698 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:44.369915 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.370589 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.371347 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.371646 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.371743 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.371979 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.372039 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.372116 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.372196 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.372258 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":38}
I20260619 14:57:44.372429 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.369106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37977 (local address 127.10.71.131:45337)
0619 14:57:44.369379 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.369395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.369505 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.369714 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.369716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.369721 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:44.369825 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.369829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.370689 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.370691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.371246 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.371254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.371751 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.371753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.371845 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.371880 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.372127 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.372131 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.372178 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.372249 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":31,"thread_start_us":100,"threads_started":1}
I20260619 14:57:44.378290 18367 raft_consensus.cc:1261] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Refusing update from remote peer ac049f724bf74d0a8b336a86a34d8d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.378618 18227 raft_consensus.cc:1261] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Refusing update from remote peer ac049f724bf74d0a8b336a86a34d8d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.378679 18589 consensus_queue.cc:1035] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.379011 18589 consensus_queue.cc:1035] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.417690 18070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.417826 18070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.419663 18070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Sending DeleteTablet for 3 replicas of tablet 316f621b52044ea1981d90627e457a9e
I20260619 14:57:44.420146 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f621b52044ea1981d90627e457a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:59720
I20260619 14:57:44.420240 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f621b52044ea1981d90627e457a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:54602
I20260619 14:57:44.420287 18605 tablet_replica.cc:330] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: stopping tablet replica
I20260619 14:57:44.420243 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f621b52044ea1981d90627e457a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34358
I20260619 14:57:44.420406 18606 tablet_replica.cc:330] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: stopping tablet replica
I20260619 14:57:44.420403 18605 raft_consensus.cc:2229] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.420419 18604 tablet_replica.cc:330] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: stopping tablet replica
I20260619 14:57:44.420492 18606 raft_consensus.cc:2229] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:44.420502 18605 raft_consensus.cc:2258] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.420535 18604 raft_consensus.cc:2229] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.420640 18604 raft_consensus.cc:2258] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.420641 18606 raft_consensus.cc:2258] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.421236 18605 ts_tablet_manager.cc:1905] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.421331 18606 ts_tablet_manager.cc:1905] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.423599 18604 ts_tablet_manager.cc:1905] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.423673 18605 ts_tablet_manager.cc:1918] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:44.423730 18605 log.cc:1196] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/316f621b52044ea1981d90627e457a9e
I20260619 14:57:44.423975 18605 ts_tablet_manager.cc:1939] T 316f621b52044ea1981d90627e457a9e P 754bbe55cd414778b19ef8b49f63b024: Deleting consensus metadata
I20260619 14:57:44.424257 18606 ts_tablet_manager.cc:1918] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:44.424346 18606 log.cc:1196] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/316f621b52044ea1981d90627e457a9e
I20260619 14:57:44.424614 18606 ts_tablet_manager.cc:1939] T 316f621b52044ea1981d90627e457a9e P ac049f724bf74d0a8b336a86a34d8d98: Deleting consensus metadata
I20260619 14:57:44.425091 18055 catalog_manager.cc:4841] TS 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661): tablet 316f621b52044ea1981d90627e457a9e (table test-table [id=e22f5077d7ea4b56861229294eba28b2]) successfully deleted
I20260619 14:57:44.426240 18056 catalog_manager.cc:4841] TS ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977): tablet 316f621b52044ea1981d90627e457a9e (table test-table [id=e22f5077d7ea4b56861229294eba28b2]) successfully deleted
I20260619 14:57:44.426554 18604 ts_tablet_manager.cc:1918] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:44.426636 18604 log.cc:1196] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/316f621b52044ea1981d90627e457a9e
I20260619 14:57:44.426882 18604 ts_tablet_manager.cc:1939] T 316f621b52044ea1981d90627e457a9e P 0e1176ed6f184e8592d86ad185919f14: Deleting consensus metadata
I20260619 14:57:44.427605 18054 catalog_manager.cc:4841] TS 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677): tablet 316f621b52044ea1981d90627e457a9e (table test-table [id=e22f5077d7ea4b56861229294eba28b2]) successfully deleted
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:57:44.449966 18617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:44.443790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.10.71.190:37049)
0619 14:57:44.443950 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.443955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.444170 (+ 215us) server_negotiation.cc:407] Connection header received
0619 14:57:44.444358 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.444361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.444417 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.444524 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:44.445230 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.445730 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.446477 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.446718 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.446822 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.449859 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59830: BlockingRecv error: recv got EOF from 127.0.0.1:59830 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM
I20260619 14:57:44.459746 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.451365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.10.71.190:37049)
0619 14:57:44.451473 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.451477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.451582 (+ 105us) server_negotiation.cc:407] Connection header received
0619 14:57:44.451681 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.451682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.451711 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.451765 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:44.452390 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.452813 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.453780 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.453954 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.454096 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.456449 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:44.456453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:44.456457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:44.456505 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:44.458149 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:44.458560 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:44.458564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:44.458565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:44.458651 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:44.458907 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:44.458910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:44.458911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:44.459091 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:44.459178 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.459496 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.459609 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:57:44.462155 18070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:57:44.462546 18070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:44.467202 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4480a4fc584524b7344adfccc3355d (DEFAULT_TABLE table=test-table [id=ce02792ecf374a6997e26b0d102a6827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.467342 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4480a4fc584524b7344adfccc3355d (DEFAULT_TABLE table=test-table [id=ce02792ecf374a6997e26b0d102a6827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.467476 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4480a4fc584524b7344adfccc3355d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.467581 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4480a4fc584524b7344adfccc3355d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.469362 18581 tablet_bootstrap.cc:492] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Bootstrap starting.
I20260619 14:57:44.469583 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4480a4fc584524b7344adfccc3355d (DEFAULT_TABLE table=test-table [id=ce02792ecf374a6997e26b0d102a6827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.469709 18582 tablet_bootstrap.cc:492] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Bootstrap starting.
I20260619 14:57:44.469842 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4480a4fc584524b7344adfccc3355d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.470204 18581 tablet_bootstrap.cc:654] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.470506 18582 tablet_bootstrap.cc:654] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.471963 18582 tablet_bootstrap.cc:492] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: No bootstrap required, opened a new log
I20260619 14:57:44.472046 18582 ts_tablet_manager.cc:1397] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:57:44.472508 18582 raft_consensus.cc:348] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.472615 18582 raft_consensus.cc:374] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.472640 18582 raft_consensus.cc:729] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754bbe55cd414778b19ef8b49f63b024, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.472735 18582 consensus_queue.cc:260] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.472929 18582 ts_tablet_manager.cc:1428] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:44.473109 18585 tablet_bootstrap.cc:492] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Bootstrap starting.
I20260619 14:57:44.473805 18585 tablet_bootstrap.cc:654] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.474884 18581 tablet_bootstrap.cc:492] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: No bootstrap required, opened a new log
I20260619 14:57:44.474998 18581 ts_tablet_manager.cc:1397] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 14:57:44.475406 18585 tablet_bootstrap.cc:492] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: No bootstrap required, opened a new log
I20260619 14:57:44.475471 18585 ts_tablet_manager.cc:1397] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:44.475555 18581 raft_consensus.cc:348] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.475659 18581 raft_consensus.cc:374] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.475688 18581 raft_consensus.cc:729] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1176ed6f184e8592d86ad185919f14, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.475791 18581 consensus_queue.cc:260] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.475805 18585 raft_consensus.cc:348] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.475885 18585 raft_consensus.cc:374] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.475924 18585 raft_consensus.cc:729] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.476034 18581 ts_tablet_manager.cc:1428] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:44.476015 18585 consensus_queue.cc:260] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.476235 18585 ts_tablet_manager.cc:1428] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:44.544512 18588 raft_consensus.cc:482] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:44.544637 18588 raft_consensus.cc:504] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.545708 18588 leader_election.cc:288] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677), ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977)
I20260619 14:57:44.550036 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.546011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:37371 (local address 127.10.71.130:36677)
0619 14:57:44.546263 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.546268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.546284 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:57:44.546409 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.546416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.546473 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.546556 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:44.546876 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.547589 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.548852 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.549450 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.549550 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.549589 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.549805 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.549857 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260619 14:57:44.550140 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.545906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:36677 (local address 127.10.71.129:37371)
0619 14:57:44.546217 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.546228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.546306 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.546593 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.546596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.546614 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:44.546754 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.546761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.547716 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.547719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.548742 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.548752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.549561 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.549565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.549672 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.549696 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.549772 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.549912 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20260619 14:57:44.550424 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.545995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:55705 (local address 127.10.71.131:37977)
0619 14:57:44.546134 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.546139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.546480 (+ 341us) server_negotiation.cc:407] Connection header received
0619 14:57:44.546607 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.546610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.546654 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.546732 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:44.547383 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.548002 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.549299 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.549903 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.550012 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.550062 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.550266 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.550331 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:57:44.550441 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.545906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37977 (local address 127.10.71.129:55705)
0619 14:57:44.546387 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.546399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.546488 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.546748 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.546750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.546763 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:44.547255 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.547264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.548110 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.548113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.549171 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.549180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.549997 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.550000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.550105 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.550133 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.550241 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.550303 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":205,"thread_start_us":57,"threads_started":1}
I20260619 14:57:44.550588 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4480a4fc584524b7344adfccc3355d" candidate_uuid: "754bbe55cd414778b19ef8b49f63b024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14" is_pre_election: true
I20260619 14:57:44.550706 18366 raft_consensus.cc:2454] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754bbe55cd414778b19ef8b49f63b024 in term 0.
I20260619 14:57:44.550849 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4480a4fc584524b7344adfccc3355d" candidate_uuid: "754bbe55cd414778b19ef8b49f63b024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac049f724bf74d0a8b336a86a34d8d98" is_pre_election: true
I20260619 14:57:44.551080 18507 raft_consensus.cc:2454] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754bbe55cd414778b19ef8b49f63b024 in term 0.
I20260619 14:57:44.551152 18158 leader_election.cc:302] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, 754bbe55cd414778b19ef8b49f63b024; no voters:
I20260619 14:57:44.551403 18588 raft_consensus.cc:2781] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:44.551478 18588 raft_consensus.cc:482] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:44.551529 18588 raft_consensus.cc:3037] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.552439 18588 raft_consensus.cc:504] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.552789 18588 leader_election.cc:288] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [CANDIDATE]: Term 1 election: Requested vote from peers 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677), ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977)
I20260619 14:57:44.553047 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4480a4fc584524b7344adfccc3355d" candidate_uuid: "754bbe55cd414778b19ef8b49f63b024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1176ed6f184e8592d86ad185919f14"
I20260619 14:57:44.553104 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4480a4fc584524b7344adfccc3355d" candidate_uuid: "754bbe55cd414778b19ef8b49f63b024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac049f724bf74d0a8b336a86a34d8d98"
I20260619 14:57:44.553174 18366 raft_consensus.cc:3037] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.553213 18507 raft_consensus.cc:3037] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.554039 18366 raft_consensus.cc:2454] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754bbe55cd414778b19ef8b49f63b024 in term 1.
I20260619 14:57:44.554049 18507 raft_consensus.cc:2454] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754bbe55cd414778b19ef8b49f63b024 in term 1.
I20260619 14:57:44.554338 18158 leader_election.cc:302] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, 754bbe55cd414778b19ef8b49f63b024; no voters:
I20260619 14:57:44.554526 18588 raft_consensus.cc:2781] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:44.554751 18588 raft_consensus.cc:686] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 1 LEADER]: Becoming Leader. State: Replica: 754bbe55cd414778b19ef8b49f63b024, State: Running, Role: LEADER
I20260619 14:57:44.555018 18588 consensus_queue.cc:237] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.557315 18069 catalog_manager.cc:5495] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129). New cstate: current_term: 1 leader_uuid: "754bbe55cd414778b19ef8b49f63b024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:44.569869 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.565894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.10.71.129:33661)
0619 14:57:44.566256 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.566268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.566288 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:57:44.566385 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.566389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.566466 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.566601 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:44.566864 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.567355 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.568274 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.568580 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.568703 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.569019 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.569437 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.569577 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.569667 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.569721 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260619 14:57:44.571885 18366 raft_consensus.cc:1261] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Refusing update from remote peer 754bbe55cd414778b19ef8b49f63b024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.572096 18507 raft_consensus.cc:1261] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Refusing update from remote peer 754bbe55cd414778b19ef8b49f63b024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.572364 18651 consensus_queue.cc:1035] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.572629 18588 consensus_queue.cc:1035] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.581794 18068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.581915 18068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.583643 18068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Sending DeleteTablet for 3 replicas of tablet 4c4480a4fc584524b7344adfccc3355d
I20260619 14:57:44.584244 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4480a4fc584524b7344adfccc3355d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:59720
I20260619 14:57:44.584344 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4480a4fc584524b7344adfccc3355d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:54602
I20260619 14:57:44.584493 18605 tablet_replica.cc:330] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: stopping tablet replica
I20260619 14:57:44.584539 18606 tablet_replica.cc:330] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: stopping tablet replica
I20260619 14:57:44.584616 18605 raft_consensus.cc:2229] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:44.584649 18606 raft_consensus.cc:2229] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.584765 18606 raft_consensus.cc:2258] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.584827 18605 raft_consensus.cc:2258] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.584959 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4480a4fc584524b7344adfccc3355d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34358
I20260619 14:57:44.585143 18604 tablet_replica.cc:330] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: stopping tablet replica
I20260619 14:57:44.585251 18604 raft_consensus.cc:2229] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.585391 18604 raft_consensus.cc:2258] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.585631 18605 ts_tablet_manager.cc:1905] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.586203 18604 ts_tablet_manager.cc:1905] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.587533 18605 ts_tablet_manager.cc:1918] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:44.587625 18605 log.cc:1196] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/4c4480a4fc584524b7344adfccc3355d
I20260619 14:57:44.587914 18605 ts_tablet_manager.cc:1939] T 4c4480a4fc584524b7344adfccc3355d P 754bbe55cd414778b19ef8b49f63b024: Deleting consensus metadata
I20260619 14:57:44.588567 18055 catalog_manager.cc:4841] TS 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661): tablet 4c4480a4fc584524b7344adfccc3355d (table test-table [id=ce02792ecf374a6997e26b0d102a6827]) successfully deleted
I20260619 14:57:44.588652 18604 ts_tablet_manager.cc:1918] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:44.588737 18604 log.cc:1196] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/4c4480a4fc584524b7344adfccc3355d
I20260619 14:57:44.589010 18604 ts_tablet_manager.cc:1939] T 4c4480a4fc584524b7344adfccc3355d P 0e1176ed6f184e8592d86ad185919f14: Deleting consensus metadata
I20260619 14:57:44.589308 18068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:57:44.589684 18054 catalog_manager.cc:4841] TS 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677): tablet 4c4480a4fc584524b7344adfccc3355d (table test-table [id=ce02792ecf374a6997e26b0d102a6827]) successfully deleted
W20260619 14:57:44.589695 18068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:57:44.589983 18606 ts_tablet_manager.cc:1905] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.592341 18606 ts_tablet_manager.cc:1918] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:57:44.592427 18606 log.cc:1196] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/4c4480a4fc584524b7344adfccc3355d
I20260619 14:57:44.592675 18606 ts_tablet_manager.cc:1939] T 4c4480a4fc584524b7344adfccc3355d P ac049f724bf74d0a8b336a86a34d8d98: Deleting consensus metadata
I20260619 14:57:44.593344 18056 catalog_manager.cc:4841] TS ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977): tablet 4c4480a4fc584524b7344adfccc3355d (table test-table [id=ce02792ecf374a6997e26b0d102a6827]) successfully deleted
I20260619 14:57:44.593783 18347 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3f2b22344c44b2a047c1fb507ac666 (DEFAULT_TABLE table=test-table [id=da20b88d6035467a82f526519e77aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.593842 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3f2b22344c44b2a047c1fb507ac666 (DEFAULT_TABLE table=test-table [id=da20b88d6035467a82f526519e77aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.594046 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3f2b22344c44b2a047c1fb507ac666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.594097 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3f2b22344c44b2a047c1fb507ac666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.595649 18585 tablet_bootstrap.cc:492] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Bootstrap starting.
I20260619 14:57:44.595793 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3f2b22344c44b2a047c1fb507ac666 (DEFAULT_TABLE table=test-table [id=da20b88d6035467a82f526519e77aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:57:44.596002 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3f2b22344c44b2a047c1fb507ac666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:44.596417 18585 tablet_bootstrap.cc:654] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.597385 18585 tablet_bootstrap.cc:492] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: No bootstrap required, opened a new log
I20260619 14:57:44.597462 18585 ts_tablet_manager.cc:1397] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 14:57:44.597811 18585 raft_consensus.cc:348] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.597911 18585 raft_consensus.cc:374] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.597950 18585 raft_consensus.cc:729] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac049f724bf74d0a8b336a86a34d8d98, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.598090 18581 tablet_bootstrap.cc:492] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Bootstrap starting.
I20260619 14:57:44.598058 18585 consensus_queue.cc:260] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.598287 18585 ts_tablet_manager.cc:1428] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:57:44.598922 18581 tablet_bootstrap.cc:654] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.599449 18582 tablet_bootstrap.cc:492] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024: Bootstrap starting.
I20260619 14:57:44.599928 18581 tablet_bootstrap.cc:492] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: No bootstrap required, opened a new log
I20260619 14:57:44.600004 18581 ts_tablet_manager.cc:1397] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:44.600093 18582 tablet_bootstrap.cc:654] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:44.600445 18581 raft_consensus.cc:348] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.600535 18581 raft_consensus.cc:374] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.600565 18581 raft_consensus.cc:729] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1176ed6f184e8592d86ad185919f14, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.600665 18581 consensus_queue.cc:260] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.600898 18581 ts_tablet_manager.cc:1428] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:57:44.601824 18582 tablet_bootstrap.cc:492] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024: No bootstrap required, opened a new log
I20260619 14:57:44.601902 18582 ts_tablet_manager.cc:1397] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:57:44.602207 18582 raft_consensus.cc:348] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.602280 18582 raft_consensus.cc:374] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:44.602298 18582 raft_consensus.cc:729] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754bbe55cd414778b19ef8b49f63b024, State: Initialized, Role: FOLLOWER
I20260619 14:57:44.602401 18582 consensus_queue.cc:260] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.602618 18582 ts_tablet_manager.cc:1428] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:57:44.770646 18586 raft_consensus.cc:482] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:57:44.770797 18586 raft_consensus.cc:504] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.771842 18586 leader_election.cc:288] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661), ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977)
I20260619 14:57:44.775478 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.772131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:55905 (local address 127.10.71.129:33661)
0619 14:57:44.772261 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.772266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.772399 (+ 133us) server_negotiation.cc:407] Connection header received
0619 14:57:44.772550 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.772553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.772604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.772686 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:44.772945 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.773414 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.774498 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.774901 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.775032 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.775101 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.775280 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.775354 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:57:44.775493 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.772203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56727 (local address 127.10.71.131:37977)
0619 14:57:44.772341 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.772345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.772508 (+ 163us) server_negotiation.cc:407] Connection header received
0619 14:57:44.772662 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.772668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.772704 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.772782 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:57:44.773069 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.773499 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.774526 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.774909 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.775036 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.775105 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.775313 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.775375 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:57:44.775552 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.771999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33661 (local address 127.10.71.130:55905)
0619 14:57:44.772312 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.772325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.772442 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.772698 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.772701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.772714 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:44.772848 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.772853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.773532 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.773535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.774375 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.774383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.775018 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.775022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.775134 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.775156 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.775243 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.775330 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":38,"thread_start_us":82,"threads_started":1}
I20260619 14:57:44.775568 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.772085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37977 (local address 127.10.71.130:56727)
0619 14:57:44.772405 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.772415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.772544 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.772793 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.772796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.772811 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:57:44.772960 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.772964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.773623 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.773626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.774401 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.774410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.775030 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.775034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.775164 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.775187 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.775287 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.775364 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":59,"thread_start_us":59,"threads_started":1}
I20260619 14:57:44.776048 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f2b22344c44b2a047c1fb507ac666" candidate_uuid: "0e1176ed6f184e8592d86ad185919f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac049f724bf74d0a8b336a86a34d8d98" is_pre_election: true
I20260619 14:57:44.776189 18507 raft_consensus.cc:2454] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1176ed6f184e8592d86ad185919f14 in term 0.
I20260619 14:57:44.776376 18226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f2b22344c44b2a047c1fb507ac666" candidate_uuid: "0e1176ed6f184e8592d86ad185919f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024" is_pre_election: true
I20260619 14:57:44.776510 18226 raft_consensus.cc:2454] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1176ed6f184e8592d86ad185919f14 in term 0.
I20260619 14:57:44.776520 18300 leader_election.cc:302] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, ac049f724bf74d0a8b336a86a34d8d98; no voters:
I20260619 14:57:44.776796 18586 raft_consensus.cc:2781] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:57:44.776886 18586 raft_consensus.cc:482] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:57:44.776939 18586 raft_consensus.cc:3037] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.777842 18586 raft_consensus.cc:504] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.778189 18586 leader_election.cc:288] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [CANDIDATE]: Term 1 election: Requested vote from peers 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661), ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977)
I20260619 14:57:44.778419 18226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f2b22344c44b2a047c1fb507ac666" candidate_uuid: "0e1176ed6f184e8592d86ad185919f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754bbe55cd414778b19ef8b49f63b024"
I20260619 14:57:44.778492 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f2b22344c44b2a047c1fb507ac666" candidate_uuid: "0e1176ed6f184e8592d86ad185919f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac049f724bf74d0a8b336a86a34d8d98"
I20260619 14:57:44.778592 18226 raft_consensus.cc:3037] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.778621 18507 raft_consensus.cc:3037] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:44.779490 18226 raft_consensus.cc:2454] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1176ed6f184e8592d86ad185919f14 in term 1.
I20260619 14:57:44.779495 18507 raft_consensus.cc:2454] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1176ed6f184e8592d86ad185919f14 in term 1.
I20260619 14:57:44.779784 18299 leader_election.cc:302] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1176ed6f184e8592d86ad185919f14, 754bbe55cd414778b19ef8b49f63b024; no voters:
I20260619 14:57:44.779963 18586 raft_consensus.cc:2781] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:44.780169 18586 raft_consensus.cc:686] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1176ed6f184e8592d86ad185919f14, State: Running, Role: LEADER
I20260619 14:57:44.780457 18586 consensus_queue.cc:237] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } }
I20260619 14:57:44.782711 18070 catalog_manager.cc:5495] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130). New cstate: current_term: 1 leader_uuid: "0e1176ed6f184e8592d86ad185919f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1176ed6f184e8592d86ad185919f14" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 } health_report { overall_health: UNKNOWN } } }
I20260619 14:57:44.796489 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.793351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.10.71.130:36677)
0619 14:57:44.793449 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.793452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.793513 (+ 61us) server_negotiation.cc:407] Connection header received
0619 14:57:44.793624 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.793626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.793685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.793765 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:44.793950 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.794418 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.795163 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.795369 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.795432 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.795688 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.796072 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.796169 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.796352 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.796407 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:57:44.805757 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.802525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:37477 (local address 127.10.71.131:37977)
0619 14:57:44.802651 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.802655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.802784 (+ 129us) server_negotiation.cc:407] Connection header received
0619 14:57:44.802923 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.802926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.802974 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.803046 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:44.803303 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.803790 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.804717 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.804942 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.805051 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.805404 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.805466 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.805520 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.805623 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.805661 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:57:44.805979 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.802402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37977 (local address 127.10.71.130:37477)
0619 14:57:44.802708 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.802719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.802832 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.803084 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.803086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.803093 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:44.803209 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.803212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.803894 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.803899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.804602 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.804611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.805062 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.805065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.805193 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.805251 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.805543 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.805548 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.805600 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.805658 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260619 14:57:44.811335 18226 raft_consensus.cc:1261] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Refusing update from remote peer 0e1176ed6f184e8592d86ad185919f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.811416 18507 raft_consensus.cc:1261] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Refusing update from remote peer 0e1176ed6f184e8592d86ad185919f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:57:44.811884 18586 consensus_queue.cc:1035] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754bbe55cd414778b19ef8b49f63b024" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.812288 18663 consensus_queue.cc:1035] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac049f724bf74d0a8b336a86a34d8d98" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:57:44.827726 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.824965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:45405 (local address 127.10.71.130:36677)
0619 14:57:44.825065 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:44.825069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:44.825083 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:44.825215 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:44.825217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:44.825265 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:44.825350 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:57:44.825539 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.826146 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.826882 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.827076 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.827160 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.827352 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.827419 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.827503 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:44.827582 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 14:57:44.827622 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:57:44.827750 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.824918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:36677 (local address 127.10.71.131:45405)
0619 14:57:44.825026 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.825040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:44.825116 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:44.825356 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:44.825358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:44.825365 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:57:44.825448 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.825453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.826259 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.826261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.826786 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:44.826793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:44.827177 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:44.827179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:44.827233 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:44.827255 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:57:44.827507 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:57:44.827515 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:57:44.827573 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:57:44.827650 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:57:44.845397 18069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.845562 18069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:57:44.847395 18069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f633206119104ec3898666356168e987: Sending DeleteTablet for 3 replicas of tablet 3e3f2b22344c44b2a047c1fb507ac666
I20260619 14:57:44.847863 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3f2b22344c44b2a047c1fb507ac666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:59720
I20260619 14:57:44.848240 18347 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3f2b22344c44b2a047c1fb507ac666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34358
I20260619 14:57:44.848429 18605 tablet_replica.cc:330] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024: stopping tablet replica
I20260619 14:57:44.848505 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18142
I20260619 14:57:44.848547 18605 raft_consensus.cc:2229] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.848650 18605 raft_consensus.cc:2258] T 3e3f2b22344c44b2a047c1fb507ac666 P 754bbe55cd414778b19ef8b49f63b024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.848793 18604 tablet_replica.cc:330] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: stopping tablet replica
I20260619 14:57:44.848901 18604 raft_consensus.cc:2229] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:57:44.849169 18604 raft_consensus.cc:2258] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.850148 18604 ts_tablet_manager.cc:1905] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.852101 18604 ts_tablet_manager.cc:1918] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:44.852190 18604 log.cc:1196] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/3e3f2b22344c44b2a047c1fb507ac666
I20260619 14:57:44.852478 18604 ts_tablet_manager.cc:1939] T 3e3f2b22344c44b2a047c1fb507ac666 P 0e1176ed6f184e8592d86ad185919f14: Deleting consensus metadata
I20260619 14:57:44.853029 18054 catalog_manager.cc:4841] TS 0e1176ed6f184e8592d86ad185919f14 (127.10.71.130:36677): tablet 3e3f2b22344c44b2a047c1fb507ac666 (table test-table [id=da20b88d6035467a82f526519e77aad1]) successfully deleted
I20260619 14:57:44.853091 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3f2b22344c44b2a047c1fb507ac666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:57:44 UTC) from {username='oryx', principal='oryx/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:54602
I20260619 14:57:44.853253 18606 tablet_replica.cc:330] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: stopping tablet replica
I20260619 14:57:44.853392 18606 raft_consensus.cc:2229] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:57:44.853493 18606 raft_consensus.cc:2258] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:57:44.854108 18606 ts_tablet_manager.cc:1905] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:57:44.856231 18606 ts_tablet_manager.cc:1918] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:57:44.856341 18606 log.cc:1196] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/3e3f2b22344c44b2a047c1fb507ac666
I20260619 14:57:44.856607 18606 ts_tablet_manager.cc:1939] T 3e3f2b22344c44b2a047c1fb507ac666 P ac049f724bf74d0a8b336a86a34d8d98: Deleting consensus metadata
I20260619 14:57:44.857220 18056 catalog_manager.cc:4841] TS ac049f724bf74d0a8b336a86a34d8d98 (127.10.71.131:37977): tablet 3e3f2b22344c44b2a047c1fb507ac666 (table test-table [id=da20b88d6035467a82f526519e77aad1]) successfully deleted
W20260619 14:57:44.858165 18055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:33661 (error 108)
I20260619 14:57:44.858345 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18282
I20260619 14:57:44.860015 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:44.859666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33661 (local address 127.0.0.1:59740)
0619 14:57:44.859861 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:44.859929 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
W20260619 14:57:44.860208 18055 catalog_manager.cc:4584] TS 754bbe55cd414778b19ef8b49f63b024 (127.10.71.129:33661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e3f2b22344c44b2a047c1fb507ac666: Network error: Client connection negotiation failed: client connection to 127.10.71.129:33661: connect: Connection refused (error 111)
I20260619 14:57:44.863327 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18422
I20260619 14:57:44.869803 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18038
2026-06-19T14:57:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3050 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18676](info): set up 2 sockets
Jun 19 14:57:44 dist-test-slave-3qdh krb5kdc[18676](info): commencing operation
krb5kdc: starting...
W20260619 14:57:45.816915 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:45 dist-test-slave-3qdh krb5kdc[18676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:45.918730 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38057
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:46.034051 18689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:46.034287 18689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:46.034359 18689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:46.038028 18689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:46.038072 18689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:46.038089 18689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:46.038128 18689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:46.038154 18689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:46.042874 18689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:38057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:46.043731 18689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:46.044678 18689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:46.052063 18695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:46.052119 18689 server_base.cc:1034] running on GCE node
W20260619 14:57:46.052076 18694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:46.052111 18697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:46.052788 18689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:46.053690 18689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:46.054855 18689 hybrid_clock.cc:648] HybridClock initialized: now 1781881066054839 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:57:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1194 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:57:46 dist-test-slave-3qdh krb5kdc[18702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:57:46 dist-test-slave-3qdh krb5kdc[18702](info): set up 2 sockets
Jun 19 14:57:46 dist-test-slave-3qdh krb5kdc[18702](info): commencing operation
krb5kdc: starting...
W20260619 14:57:47.008773 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:57:47.111016 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37699
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:37699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:47.208345 18715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:47.208544 18715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:47.208580 18715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:47.211850 18715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:47.211895 18715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:47.211912 18715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:47.211926 18715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:47.211939 18715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:47.216566 18715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:37699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:47.217236 18715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:47.217931 18715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:47.224668 18723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.224684 18715 server_base.cc:1034] running on GCE node
W20260619 14:57:47.224740 18721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:47.224642 18720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.225281 18715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:47.226182 18715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:47.227370 18715 hybrid_clock.cc:648] HybridClock initialized: now 1781881067227348 us; error 32 us; skew 500 ppm
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:47.233386 18715 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:47.234112 18715 webserver.cc:460] Webserver started at http://127.10.71.190:43561/ using document root <none> and password file <none>
I20260619 14:57:47.234615 18715 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:47.234671 18715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:47.234876 18715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:47.236582 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "daec8d64978547e2abfcb55194871860"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "33c86c1fb9450ae77b5e5445acadac59"
server_key_iv: "36f7cada83e4207bd6304f966e839510"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.236984 18715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "daec8d64978547e2abfcb55194871860"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "33c86c1fb9450ae77b5e5445acadac59"
server_key_iv: "36f7cada83e4207bd6304f966e839510"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.239761 18715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:47.241726 18730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.242727 18715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:57:47.242857 18715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "daec8d64978547e2abfcb55194871860"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "33c86c1fb9450ae77b5e5445acadac59"
server_key_iv: "36f7cada83e4207bd6304f966e839510"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.242949 18715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:47.255263 18715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:47.257489 18715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:47.257669 18715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:47.263764 18715 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:37699
I20260619 14:57:47.263772 18782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:37699 every 8 connection(s)
I20260619 14:57:47.264648 18715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:47.265535 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18715
I20260619 14:57:47.265666 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:57:47.265908 10526 external_mini_cluster.cc:1428] Setting key 19e24635936f20cd51747e6f86878673
I20260619 14:57:47.267309 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:57:47.273279 18783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Bootstrap starting.
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:47.275573 18783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Neither blocks nor log segments found. Creating new log.
I20260619 14:57:47.276196 18783 log.cc:824] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:47.277982 18783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: No bootstrap required, opened a new log
I20260619 14:57:47.279433 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.267064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.10.71.190:37699)
0619 14:57:47.267450 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:47.267465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:47.267500 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:57:47.268248 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:47.268268 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:47.268664 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:47.268981 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:47.269611 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.270970 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.271916 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.272639 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.272795 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.275265 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:47.275269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:47.275279 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:47.275303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:47.277031 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.277435 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.277438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.277442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.277527 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.277781 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.277783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.277784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.278101 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:47.278235 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:47.278487 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 14:57:47.278732 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260619 14:57:47.280809 18783 raft_consensus.cc:348] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:47.281025 18783 raft_consensus.cc:374] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:57:47.281078 18783 raft_consensus.cc:729] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daec8d64978547e2abfcb55194871860, State: Initialized, Role: FOLLOWER
I20260619 14:57:47.281440 18783 consensus_queue.cc:260] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:47.281544 18783 raft_consensus.cc:388] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:47.281591 18783 raft_consensus.cc:482] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:47.281651 18783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:57:47.282286 18783 raft_consensus.cc:504] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:47.282593 18783 leader_election.cc:302] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daec8d64978547e2abfcb55194871860; no voters:
I20260619 14:57:47.282866 18783 leader_election.cc:288] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:57:47.282951 18788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:57:47.283213 18788 raft_consensus.cc:686] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 LEADER]: Becoming Leader. State: Replica: daec8d64978547e2abfcb55194871860, State: Running, Role: LEADER
I20260619 14:57:47.283437 18788 consensus_queue.cc:237] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:47.283864 18783 sys_catalog.cc:564] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:47.284822 18790 sys_catalog.cc:455] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daec8d64978547e2abfcb55194871860. Latest consensus state: current_term: 1 leader_uuid: "daec8d64978547e2abfcb55194871860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } } }
I20260619 14:57:47.284948 18790 sys_catalog.cc:458] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:47.285221 18789 sys_catalog.cc:455] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daec8d64978547e2abfcb55194871860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } } }
I20260619 14:57:47.285315 18789 sys_catalog.cc:458] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: This master's current role is: LEADER
W20260619 14:57:47.288354 18804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:57:47.288439 18804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:57:47.288514 18797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:47.289279 18797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:47.295506 18797 catalog_manager.cc:1347] Generated new cluster ID: 3b710ea589ff4e7e994b30934f4aebf1
I20260619 14:57:47.295585 18797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:47.302989 18797 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:57:47.303695 18797 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:47.308486 18797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Generated new TSK 0
I20260619 14:57:47.309075 18797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:57:47.340068 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37699
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:47.434453 18809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:47.434660 18809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:47.434737 18809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:47.438436 18809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:47.438478 18809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:47.438556 18809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:47.442763 18809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:47.443521 18809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:47.444439 18809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:57:47.452505 18809 server_base.cc:1034] running on GCE node
W20260619 14:57:47.452477 18815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:47.452490 18817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:47.452478 18814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.453033 18809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:47.453724 18809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:47.454918 18809 hybrid_clock.cc:648] HybridClock initialized: now 1781881067454900 us; error 42 us; skew 500 ppm
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:47.457374 18809 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:57:47.458071 18809 webserver.cc:460] Webserver started at http://127.10.71.129:42821/ using document root <none> and password file <none>
I20260619 14:57:47.458518 18809 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:47.458606 18809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:47.458799 18809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:47.460479 18809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "edbc945e8bbe41a6a485beb09846cd8e"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "ec4580f977efbb28ae8ff0bdc099da0d"
server_key_iv: "25d1f387afd63fcecd71893342f1a3cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.460975 18809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "edbc945e8bbe41a6a485beb09846cd8e"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "ec4580f977efbb28ae8ff0bdc099da0d"
server_key_iv: "25d1f387afd63fcecd71893342f1a3cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.463944 18809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:57:47.466146 18824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.467262 18809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:47.467389 18809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "edbc945e8bbe41a6a485beb09846cd8e"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "ec4580f977efbb28ae8ff0bdc099da0d"
server_key_iv: "25d1f387afd63fcecd71893342f1a3cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.467487 18809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:47.478710 18809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:47.480967 18809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:47.481144 18809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:47.481616 18809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:47.482280 18809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:47.482353 18809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.482409 18809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:47.482468 18809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.491170 18809 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:43807
I20260619 14:57:47.491192 18937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:43807 every 8 connection(s)
I20260619 14:57:47.491875 18809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:47.495105 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18809
I20260619 14:57:47.495252 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:57:47.495622 10526 external_mini_cluster.cc:1428] Setting key c66faad35dc5910284a5da97eab3f027
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:57:47.504901 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.493451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:34109 (local address 127.10.71.190:37699)
0619 14:57:47.493569 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:47.493573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:47.494390 (+ 817us) server_negotiation.cc:407] Connection header received
0619 14:57:47.495443 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:47.495458 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:47.495507 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:47.495582 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:47.496865 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.497366 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.498804 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.499157 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.499263 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.501651 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:47.501654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:47.501656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:47.501674 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:47.502885 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.503428 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.503430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.503431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.503491 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.503849 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.503851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.503852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.503984 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:47.504045 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:47.504699 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:57:47.504793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:47.505705 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.493707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:34109)
0619 14:57:47.494227 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:47.494260 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:47.495164 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:47.495730 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:47.495737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:47.496100 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:47.496643 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:47.496660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.497490 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.497496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:47.498695 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:47.498704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.499282 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.499285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:47.499402 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.499873 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:47.499887 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:47.501491 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:47.502996 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:47.503003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:47.503016 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:47.503335 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:47.503575 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:47.503581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:47.503584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:47.503754 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:47.504093 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:47.504100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:47.504398 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 14:57:47.504618 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 14:57:47.504940 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":91,"threads_started":1}
I20260619 14:57:47.506659 18938 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:57:47.506875 18938 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:47.507239 18938 heartbeater.cc:511] Master 127.10.71.190:37699 requested a full tablet report, sending...
I20260619 14:57:47.508745 18747 ts_manager.cc:194] Registered new tserver with Master: edbc945e8bbe41a6a485beb09846cd8e (127.10.71.129:43807)
I20260619 14:57:47.510205 18747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:34109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:57:47.518656 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37699
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:47.611061 18943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:47.611238 18943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:47.611270 18943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:47.614785 18943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:47.614833 18943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:47.614933 18943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:47.619249 18943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:47.620041 18943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:47.620898 18943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:47.629565 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.629657 18943 server_base.cc:1034] running on GCE node
W20260619 14:57:47.629588 18948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:47.629609 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.630195 18943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:47.630721 18943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:47.631894 18943 hybrid_clock.cc:648] HybridClock initialized: now 1781881067631877 us; error 31 us; skew 500 ppm
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:47.634498 18943 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:57:47.635252 18943 webserver.cc:460] Webserver started at http://127.10.71.130:44677/ using document root <none> and password file <none>
I20260619 14:57:47.635701 18943 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:47.635756 18943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:47.635893 18943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:47.637573 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "3a77e2c1c1e44f11896bcf00b9203061"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "4a7beafec1f118f52bed221c26a547c1"
server_key_iv: "e99b3f516c855ef400b369356150b130"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.638017 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "3a77e2c1c1e44f11896bcf00b9203061"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "4a7beafec1f118f52bed221c26a547c1"
server_key_iv: "e99b3f516c855ef400b369356150b130"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.641150 18943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:57:47.643177 18958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.644176 18943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:57:47.644263 18943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "3a77e2c1c1e44f11896bcf00b9203061"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "4a7beafec1f118f52bed221c26a547c1"
server_key_iv: "e99b3f516c855ef400b369356150b130"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.644447 18943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:47.660528 18943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:47.663230 18943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:47.663355 18943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:47.663765 18943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:47.664505 18943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:47.664549 18943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.664588 18943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:47.664603 18943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.673056 18943 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:41889
I20260619 14:57:47.673130 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:41889 every 8 connection(s)
I20260619 14:57:47.673774 18943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:47.683175 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 18943
I20260619 14:57:47.683308 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:57:47.683681 10526 external_mini_cluster.cc:1428] Setting key 6051c0d4ebdb32df01c708360c8f6deb
I20260619 14:57:47.685899 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.675328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:38111 (local address 127.10.71.190:37699)
0619 14:57:47.675463 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:47.675467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:47.676351 (+ 884us) server_negotiation.cc:407] Connection header received
0619 14:57:47.677166 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:47.677170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:47.677245 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:47.677353 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:47.678335 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.678796 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.679784 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.679984 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.680096 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.682104 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:47.682108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:47.682110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:47.682130 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:47.683471 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.684015 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.684020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.684023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.684117 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.684617 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.684620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.684621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.684858 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:47.685019 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:47.685570 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 14:57:47.685687 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:57:47.686416 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.675662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:38111)
0619 14:57:47.676206 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:47.676240 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:47.676966 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:47.677498 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:47.677505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:47.677806 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:47.678181 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:47.678192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.678914 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.678918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:47.679659 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:47.679668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.680120 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.680125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:47.680316 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.680732 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:47.680746 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:47.681940 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:47.683618 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:47.683624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:47.683635 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:47.683921 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:47.684210 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:47.684215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:47.684219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:47.684520 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:47.684975 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:47.684981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:47.685252 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 14:57:47.685401 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 14:57:47.685712 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":89,"threads_started":1}
I20260619 14:57:47.687734 19072 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37699
I20260619 14:57:47.687973 19072 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:47.688493 19072 heartbeater.cc:511] Master 127.10.71.190:37699 requested a full tablet report, sending...
I20260619 14:57:47.689443 18747 ts_manager.cc:194] Registered new tserver with Master: 3a77e2c1c1e44f11896bcf00b9203061 (127.10.71.130:41889)
I20260619 14:57:47.689980 18747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:38111
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:57:47.709146 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37699
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:47.805757 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:47.805989 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:47.806066 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:47.809969 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:47.810048 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:47.810195 19077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:47.814792 19077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:47.815646 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:47.816468 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:47.823964 19085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:47.823946 19082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.824069 19077 server_base.cc:1034] running on GCE node
W20260619 14:57:47.824232 19083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:47.824627 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:47.825153 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:47.826337 19077 hybrid_clock.cc:648] HybridClock initialized: now 1781881067826328 us; error 23 us; skew 500 ppm
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:47.828575 19077 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:57:47.829226 19077 webserver.cc:460] Webserver started at http://127.10.71.131:38655/ using document root <none> and password file <none>
I20260619 14:57:47.829653 19077 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:47.829727 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:47.829909 19077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:57:47.831421 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "cf45ec49d31c4e7e819f2bb907e0be72"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "7a7cb9619f93b016b84720618ab00439"
server_key_iv: "8b928f925abb6104fd000968dca5536a"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.831892 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "cf45ec49d31c4e7e819f2bb907e0be72"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "7a7cb9619f93b016b84720618ab00439"
server_key_iv: "8b928f925abb6104fd000968dca5536a"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.834988 19077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:57:47.836971 19092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.837931 19077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:57:47.838042 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "cf45ec49d31c4e7e819f2bb907e0be72"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "7a7cb9619f93b016b84720618ab00439"
server_key_iv: "8b928f925abb6104fd000968dca5536a"
server_key_version: "encryptionkey@0"
I20260619 14:57:47.838137 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:47.849680 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:47.851797 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:47.851958 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:47.852460 19077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:47.853111 19077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:47.853171 19077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.853219 19077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:47.853266 19077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:47.860852 19077 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:34215
I20260619 14:57:47.860877 19205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:34215 every 8 connection(s)
I20260619 14:57:47.861495 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:47.863521 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19077
I20260619 14:57:47.863633 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:57:47.863878 10526 external_mini_cluster.cc:1428] Setting key 5056934bb5b99a3c926d0a4ba09a2e13
Jun 19 14:57:47 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:57:47.872169 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.862894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:49393 (local address 127.10.71.190:37699)
0619 14:57:47.863008 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:47.863011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:47.863895 (+ 884us) server_negotiation.cc:407] Connection header received
0619 14:57:47.864740 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:47.864743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:47.864798 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:47.864856 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:47.865893 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.866403 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.867304 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.867480 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.867544 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.869605 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:47.869608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:47.869609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:47.869629 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:47.870715 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.871128 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.871131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.871132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.871193 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:47.871437 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:47.871439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:47.871441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:47.871577 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:47.871641 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:47.871984 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 14:57:47.872075 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:57:47.872810 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:47.863138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:49393)
0619 14:57:47.863759 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:47.863805 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:47.864550 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:47.865010 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:47.865017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:47.865311 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:47.865768 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:47.865783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.866496 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.866498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:47.867215 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:47.867223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:47.867580 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:47.867582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:47.867725 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:47.868203 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:47.868218 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:47.869480 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:57:47.870800 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:47.870804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:47.870812 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:47.871045 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:47.871261 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:57:47.871263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:57:47.871264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:57:47.871369 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:57:47.871650 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:57:47.871653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:57:47.871820 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 14:57:47.871940 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:57:47.872115 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":90,"threads_started":1}
I20260619 14:57:47.873811 19206 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37699
I20260619 14:57:47.873994 19206 heartbeater.cc:464] Registering TS with master...
I20260619 14:57:47.874356 19206 heartbeater.cc:511] Master 127.10.71.190:37699 requested a full tablet report, sending...
I20260619 14:57:47.875216 18747 ts_manager.cc:194] Registered new tserver with Master: cf45ec49d31c4e7e819f2bb907e0be72 (127.10.71.131:34215)
I20260619 14:57:47.875746 18747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:49393
I20260619 14:57:47.877385 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab.
Entry for principal oryx/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab.
WARNING: no policy specified for oryx/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab.
Entry for principal oryx/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab.
WARNING: no policy specified for oryx/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab.
Entry for principal oryx/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab.
I20260619 14:57:47.934387 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18809
I20260619 14:57:47.939355 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18943
I20260619 14:57:47.944167 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19077
I20260619 14:57:47.948764 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 18715
I20260619 14:57:47.954079 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37699
--webserver_interface=127.10.71.190
--webserver_port=43561
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:37699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:48.052152 19215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:48.052395 19215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:48.052457 19215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:48.056190 19215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:57:48.056265 19215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:48.056300 19215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:48.056352 19215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:57:48.056383 19215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:57:48.060876 19215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:37699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:48.061676 19215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:48.062498 19215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:57:48.070133 19215 server_base.cc:1034] running on GCE node
W20260619 14:57:48.070188 19221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.070120 19220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.070147 19223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:48.070731 19215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:48.071635 19215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:48.072788 19215 hybrid_clock.cc:648] HybridClock initialized: now 1781881068072753 us; error 44 us; skew 500 ppm
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881068, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:48.075302 19215 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:57:48.076026 19215 webserver.cc:460] Webserver started at http://127.10.71.190:43561/ using document root <none> and password file <none>
I20260619 14:57:48.076575 19215 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:48.076680 19215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:48.080042 19215 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:48.082017 19230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.083124 19215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:57:48.083249 19215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "daec8d64978547e2abfcb55194871860"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "33c86c1fb9450ae77b5e5445acadac59"
server_key_iv: "36f7cada83e4207bd6304f966e839510"
server_key_version: "encryptionkey@0"
I20260619 14:57:48.083709 19215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:48.100538 19215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:48.102845 19215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:48.103011 19215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:48.108970 19215 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:37699
I20260619 14:57:48.108994 19282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:37699 every 8 connection(s)
I20260619 14:57:48.109807 19215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:57:48.113343 19283 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:57:48.114749 19283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Bootstrap starting.
I20260619 14:57:48.118703 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19215
I20260619 14:57:48.119182 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:43807
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:37699
--webserver_port=42821
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:57:48.129586 19283 log.cc:824] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Log is configured to *not* fsync() on all Append() calls
I20260619 14:57:48.135025 19283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:57:48.135347 19283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Bootstrap complete.
I20260619 14:57:48.138948 19283 raft_consensus.cc:348] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:48.139312 19283 raft_consensus.cc:729] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daec8d64978547e2abfcb55194871860, State: Initialized, Role: FOLLOWER
I20260619 14:57:48.139770 19283 consensus_queue.cc:260] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:48.139891 19283 raft_consensus.cc:388] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:57:48.139966 19283 raft_consensus.cc:482] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:57:48.140064 19283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:57:48.141675 19283 raft_consensus.cc:504] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:48.141955 19283 leader_election.cc:302] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daec8d64978547e2abfcb55194871860; no voters:
I20260619 14:57:48.142227 19283 leader_election.cc:288] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:57:48.142287 19287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:57:48.142472 19287 raft_consensus.cc:686] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [term 2 LEADER]: Becoming Leader. State: Replica: daec8d64978547e2abfcb55194871860, State: Running, Role: LEADER
I20260619 14:57:48.142705 19287 consensus_queue.cc:237] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } }
I20260619 14:57:48.143268 19283 sys_catalog.cc:564] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:57:48.143504 19289 sys_catalog.cc:455] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daec8d64978547e2abfcb55194871860. Latest consensus state: current_term: 2 leader_uuid: "daec8d64978547e2abfcb55194871860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } } }
I20260619 14:57:48.143530 19288 sys_catalog.cc:455] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "daec8d64978547e2abfcb55194871860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daec8d64978547e2abfcb55194871860" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37699 } } }
I20260619 14:57:48.143616 19289 sys_catalog.cc:458] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:48.143628 19288 sys_catalog.cc:458] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860 [sys.catalog]: This master's current role is: LEADER
I20260619 14:57:48.143913 19291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:57:48.146838 19291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:57:48.147418 19291 catalog_manager.cc:1259] Loaded cluster ID: 3b710ea589ff4e7e994b30934f4aebf1
I20260619 14:57:48.147497 19291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:57:48.149688 19291 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:57:48.150285 19291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P daec8d64978547e2abfcb55194871860: Loaded TSK: 0
I20260619 14:57:48.151386 19291 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:57:48.219715 19285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:48.219961 19285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:48.220028 19285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:48.223753 19285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:48.223824 19285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:48.223908 19285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:57:48.228549 19285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.129.keytab
--principal=oryx/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:43807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=42821
--tserver_master_addrs=127.10.71.190:37699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:48.229372 19285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:48.230206 19285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:48.239096 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.239166 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.239100 19312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:48.239349 19285 server_base.cc:1034] running on GCE node
I20260619 14:57:48.239766 19285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:48.240437 19285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:48.241639 19285 hybrid_clock.cc:648] HybridClock initialized: now 1781881068241613 us; error 37 us; skew 500 ppm
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881068, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:48.244894 19285 init.cc:377] Logged in from keytab as oryx/127.10.71.129@KRBTEST.COM (short username oryx)
I20260619 14:57:48.245719 19285 webserver.cc:460] Webserver started at http://127.10.71.129:42821/ using document root <none> and password file <none>
I20260619 14:57:48.246222 19285 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:48.246318 19285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:48.249712 19285 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:48.251639 19319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.252807 19285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:48.252940 19285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "edbc945e8bbe41a6a485beb09846cd8e"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "ec4580f977efbb28ae8ff0bdc099da0d"
server_key_iv: "25d1f387afd63fcecd71893342f1a3cb"
server_key_version: "encryptionkey@0"
I20260619 14:57:48.253502 19285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:48.277752 19285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:48.280104 19285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:48.280277 19285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:48.280853 19285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:48.281594 19285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:48.281638 19285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.281699 19285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:48.281730 19285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.290520 19285 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:43807
I20260619 14:57:48.290540 19432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:43807 every 8 connection(s)
I20260619 14:57:48.291324 19285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:57:48.294693 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19285
I20260619 14:57:48.295261 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:41889
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:37699
--webserver_port=44677
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.304131 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.293288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:44697)
0619 14:57:48.293858 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.293902 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.294912 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.296223 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.296233 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.296729 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.297312 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.297331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.298841 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.298843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.299573 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.299582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.300563 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.300566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.300685 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.301194 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.301209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.303467 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":114,"threads_started":1}
W20260619 14:57:48.304672 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.293152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:44697 (local address 127.10.71.190:37699)
0619 14:57:48.293503 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.293516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.294005 (+ 489us) server_negotiation.cc:407] Connection header received
0619 14:57:48.295425 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.295439 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.295800 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.296109 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.297483 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.298732 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.299690 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.300429 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.300589 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.303730 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:44697: BlockingRecv error: recv got EOF from 127.10.71.129:44697 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
W20260619 14:57:48.304728 19433 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:37699 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.71.190:37699: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.310384 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.305455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:53655)
0619 14:57:48.305578 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.305594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.305677 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.305898 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.305900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.306007 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.306099 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.306104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.307054 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.307056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.307495 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.307501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.308070 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.308073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.308138 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.308575 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.308587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.310261 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:57:48.310567 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.305554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:53655 (local address 127.10.71.190:37699)
0619 14:57:48.305687 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.305691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.305704 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:57:48.305742 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.305746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.305799 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.305884 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.306202 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.306953 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.307587 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.307950 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.308041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.310403 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:53655: BlockingRecv error: recv got EOF from 127.10.71.129:53655 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.315423 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.310925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:40757)
0619 14:57:48.311002 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.311016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.311088 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.311280 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.311282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.311385 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.311441 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.311445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.312254 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.312256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.312785 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.312792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.313272 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.313274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.313321 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.313680 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.313690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.315315 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 14:57:48.315585 19433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:57:48.315716 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.310972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:40757 (local address 127.10.71.190:37699)
0619 14:57:48.311056 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.311067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.311079 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:48.311157 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.311159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.311190 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.311256 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.311527 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.312162 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.312889 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.313179 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.313264 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.315431 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:40757: BlockingRecv error: recv got EOF from 127.10.71.129:40757 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 14:57:48.398700 19437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:48.398890 19437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:48.398927 19437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:48.402400 19437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:48.402442 19437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:48.402523 19437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:57:48.407212 19437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.130.keytab
--principal=oryx/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:41889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=44677
--tserver_master_addrs=127.10.71.190:37699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:48.407979 19437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:48.408847 19437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:57:48.416807 19437 server_base.cc:1034] running on GCE node
W20260619 14:57:48.416762 19442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.416765 19443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.416785 19445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:48.417313 19437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:48.417867 19437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:48.419059 19437 hybrid_clock.cc:648] HybridClock initialized: now 1781881068419042 us; error 27 us; skew 500 ppm
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881068, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:48.421492 19437 init.cc:377] Logged in from keytab as oryx/127.10.71.130@KRBTEST.COM (short username oryx)
I20260619 14:57:48.422156 19437 webserver.cc:460] Webserver started at http://127.10.71.130:44677/ using document root <none> and password file <none>
I20260619 14:57:48.422575 19437 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:48.422642 19437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:48.425891 19437 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:48.427525 19452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.428646 19437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:48.428730 19437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "3a77e2c1c1e44f11896bcf00b9203061"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "4a7beafec1f118f52bed221c26a547c1"
server_key_iv: "e99b3f516c855ef400b369356150b130"
server_key_version: "encryptionkey@0"
I20260619 14:57:48.429134 19437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:48.451539 19437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:48.454013 19437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:48.454133 19437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:48.454550 19437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:48.455194 19437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:48.455236 19437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.455269 19437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:48.455284 19437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.463042 19437 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:41889
I20260619 14:57:48.463081 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:41889 every 8 connection(s)
I20260619 14:57:48.463792 19437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.472079 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19437
I20260619 14:57:48.472571 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:34215
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:37699
--webserver_port=38655
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:57:48.472878 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.465290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:47817 (local address 127.10.71.190:37699)
0619 14:57:48.465453 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.465459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.466276 (+ 817us) server_negotiation.cc:407] Connection header received
0619 14:57:48.467061 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.467063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.467113 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.467185 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.468096 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.468642 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.469601 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.469787 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.469865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.472736 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:47817: BlockingRecv error: recv got EOF from 127.10.71.130:47817 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:57:48.473367 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.465641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:47817)
0619 14:57:48.466151 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.466181 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.466877 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.467310 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.467315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.467586 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.467977 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.467988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.468734 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.468737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.469490 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.469503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.469895 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.469898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.470020 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.470459 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.470472 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.472572 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":98,"threads_started":1}
W20260619 14:57:48.473783 19566 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:37699 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.71.190:37699: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:57:48.479784 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.474232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:33389 (local address 127.10.71.190:37699)
0619 14:57:48.474357 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.474364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.474378 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:48.474441 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.474446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.474490 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.474584 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.474921 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.475484 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.476663 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.476858 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.476951 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.479714 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:33389: BlockingRecv error: recv got EOF from 127.10.71.130:33389 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:57:48.479796 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.474123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:33389)
0619 14:57:48.474279 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.474293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.474366 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.474606 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.474608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.474731 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.474812 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.474817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.475631 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.475637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.476534 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.476547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.476989 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.476996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.477113 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.477754 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.477787 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.479628 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.484961 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.480267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:48381)
0619 14:57:48.480371 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.480383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.480462 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.480669 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.480672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.480816 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.480867 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.480872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.481566 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.481569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.482069 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.482076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.482499 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.482502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.482576 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.483071 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.483083 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.484858 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:57:48.485113 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.480374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:48381 (local address 127.10.71.190:37699)
0619 14:57:48.480479 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.480484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.480498 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:57:48.480530 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.480533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.480573 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.480659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.480995 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.481467 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.482180 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.482385 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.482501 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.484971 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:48381: BlockingRecv error: recv got EOF from 127.10.71.130:48381 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:57:48.485114 19566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:57:48.574361 19569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:57:48.574584 19569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:57:48.574622 19569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:57:48.578397 19569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:57:48.578439 19569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:57:48.578486 19569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:57:48.583021 19569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/krb5kdc/oryx_127.10.71.131.keytab
--principal=oryx/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:34215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=38655
--tserver_master_addrs=127.10.71.190:37699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:57:48.583806 19569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:57:48.584687 19569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:57:48.593009 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.593086 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:57:48.593008 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:57:48.593201 19569 server_base.cc:1034] running on GCE node
I20260619 14:57:48.593544 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:57:48.593997 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:57:48.595139 19569 hybrid_clock.cc:648] HybridClock initialized: now 1781881068595123 us; error 25 us; skew 500 ppm
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881068, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:57:48.597468 19569 init.cc:377] Logged in from keytab as oryx/127.10.71.131@KRBTEST.COM (short username oryx)
I20260619 14:57:48.598151 19569 webserver.cc:460] Webserver started at http://127.10.71.131:38655/ using document root <none> and password file <none>
I20260619 14:57:48.598587 19569 fs_manager.cc:359] Metadata directory not provided
I20260619 14:57:48.598678 19569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:57:48.602160 19569 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:57:48.603868 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.605069 19569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:57:48.605190 19569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "cf45ec49d31c4e7e819f2bb907e0be72"
format_stamp: "Formatted at 2026-06-19 14:57:47 on dist-test-slave-3qdh"
server_key: "7a7cb9619f93b016b84720618ab00439"
server_key_iv: "8b928f925abb6104fd000968dca5536a"
server_key_version: "encryptionkey@0"
I20260619 14:57:48.605661 19569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:57:48.623910 19569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:57:48.626369 19569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:57:48.626538 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:57:48.627002 19569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:57:48.627651 19569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:57:48.627713 19569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.627780 19569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:57:48.627835 19569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:57:48.636144 19569 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:34215
I20260619 14:57:48.636209 19697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:34215 every 8 connection(s)
I20260619 14:57:48.637022 19569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:57:48.638624 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19569
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:57:48.646953 19435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.638579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:49637 (local address 127.10.71.190:37699)
0619 14:57:48.638698 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.638702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.639692 (+ 990us) server_negotiation.cc:407] Connection header received
0619 14:57:48.640475 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.640478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.640519 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.640587 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.641498 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.641915 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.643030 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.643227 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.643339 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.646850 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:49637: BlockingRecv error: recv got EOF from 127.10.71.131:49637 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:57:48.647560 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.638889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:49637)
0619 14:57:48.639562 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.639603 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.640262 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.640732 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.640738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.641004 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.641382 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.641392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.642020 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.642023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.642906 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.642915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.643362 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.643366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.643577 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.644076 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.644098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.646733 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":120,"threads_started":1}
W20260619 14:57:48.647938 19698 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:37699 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.71.190:37699: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
I20260619 14:57:48.648102 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.639564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.10.71.190:37699)
0619 14:57:48.639791 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.639795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.639900 (+ 105us) server_negotiation.cc:407] Connection header received
0619 14:57:48.640043 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.640049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.640093 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.640176 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.640873 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.641357 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.642098 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.642314 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.642412 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.643692 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:57:48.643697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:57:48.643707 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:57:48.643730 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:57:48.645856 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:48.646397 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:48.646402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:48.646411 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:48.646545 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:57:48.646806 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:57:48.646809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:57:48.646810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:57:48.647179 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:57:48.647364 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:57:48.647610 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 14:57:48.647956 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.652532 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.648340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:35443)
0619 14:57:48.648511 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.648525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.648596 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.648796 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.648799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.648937 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.649006 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.649011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.649691 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.649696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.650242 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.650254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.650653 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.650655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.650715 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.651015 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.651024 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.652466 (+ 1442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260619 14:57:48.652637 19702 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.648413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:35443 (local address 127.10.71.190:37699)
0619 14:57:48.648506 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.648509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.648562 (+ 53us) server_negotiation.cc:407] Connection header received
0619 14:57:48.648661 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.648664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.648696 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.648788 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.649108 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.649591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.650381 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.650569 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.650640 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.652555 (+ 1915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:35443: BlockingRecv error: recv got EOF from 127.10.71.131:35443 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:48 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:48.657119 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:48.653021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:39747)
0619 14:57:48.653148 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:48.653161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:48.653240 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:48.653414 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:48.653416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:48.653523 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:48.653581 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.653585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.654169 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.654172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.654675 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:48.654682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.655037 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.655040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:48.655107 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.655515 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:48.655525 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:48.657051 (+ 1526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 14:57:48.657284 19702 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:48.653091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:39747 (local address 127.10.71.190:37699)
0619 14:57:48.653193 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:48.653195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:48.653206 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:48.653294 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:48.653296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:48.653335 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:48.653405 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:48.653663 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.654073 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.654767 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:48.654951 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:48.655021 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:48.657185 (+ 2164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:39747: BlockingRecv error: recv got EOF from 127.10.71.131:39747 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:57:48.657346 19698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:57:49 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:49 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:49.321661 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:49.316172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:56803)
0619 14:57:49.316445 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:49.316457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:49.316544 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:49.316769 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:49.316771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:49.316926 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:49.317016 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:49.317021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.317829 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.317833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:49.318800 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:49.318809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.319217 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.319220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:49.319277 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:49.319791 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:49.319805 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:49.321547 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260619 14:57:49.321915 19704 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:49.316242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:56803 (local address 127.10.71.190:37699)
0619 14:57:49.316511 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:49.316528 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:49.316556 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:57:49.316614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:49.316616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:49.316657 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:49.316739 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:49.317137 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.317696 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.318897 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.319124 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.319204 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:49.321755 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:56803: BlockingRecv error: recv got EOF from 127.10.71.129:56803 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
Jun 19 14:57:49 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:49 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:49.490696 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:49.485717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:56449)
0619 14:57:49.485940 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:49.485952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:49.486073 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:49.486323 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:49.486326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:49.486508 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:49.486588 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:49.486593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.487339 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.487342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:49.487861 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:49.487868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.488340 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.488343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:49.488446 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:49.488920 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:49.488930 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:49.490597 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":62,"threads_started":1}
W20260619 14:57:49.490813 19704 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:49.485769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56449 (local address 127.10.71.190:37699)
0619 14:57:49.485908 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:49.485912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:49.486030 (+ 118us) server_negotiation.cc:407] Connection header received
0619 14:57:49.486169 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:49.486172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:49.486221 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:49.486308 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:49.486691 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.487230 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.487975 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.488213 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.488333 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:49.490712 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:56449: BlockingRecv error: recv got EOF from 127.10.71.130:56449 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:57:49 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:49 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:49.662853 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:49.657824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:43899)
0619 14:57:49.658091 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:49.658103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:49.658186 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:49.658390 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:49.658393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:49.658545 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:49.658670 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:49.658675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.659396 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.659398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:49.659905 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:49.659912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.660325 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.660327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:49.660403 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:49.660975 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:49.660985 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:49.662714 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260619 14:57:49.662909 19704 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:49.657948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:43899 (local address 127.10.71.190:37699)
0619 14:57:49.658072 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:49.658076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:49.658162 (+ 86us) server_negotiation.cc:407] Connection header received
0619 14:57:49.658248 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:49.658250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:49.658292 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:49.658359 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:49.658781 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.659308 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.659997 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:49.660216 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:49.660291 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:49.662811 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:43899: BlockingRecv error: recv got EOF from 127.10.71.131:43899 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:57:50 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:50 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:50.327109 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:50.322370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:55301)
0619 14:57:50.322679 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:50.322693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:50.322776 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:50.322984 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:50.322986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:50.323156 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:50.323240 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:50.323245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.323891 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.323892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:50.324395 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:50.324402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.324825 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.324828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:50.324891 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:50.325432 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:50.325444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:50.326989 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
W20260619 14:57:50.327260 19708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:50.322495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:55301 (local address 127.10.71.190:37699)
0619 14:57:50.322747 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:50.322750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:50.322762 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:50.322843 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:50.322845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:50.322883 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:50.322952 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:50.323330 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.323813 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.324483 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.324732 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.324811 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:50.327115 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:55301: BlockingRecv error: recv got EOF from 127.10.71.129:55301 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
Jun 19 14:57:50 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:50 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:50.496455 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:50.491358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:46991)
0619 14:57:50.491616 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:50.491628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:50.491709 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:50.491924 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:50.491926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:50.492072 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:50.492190 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:50.492194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.493002 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.493005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:50.493554 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:50.493562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.493951 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.493954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:50.494013 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:50.494475 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:50.494486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:50.496282 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260619 14:57:50.496552 19708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:50.491460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:46991 (local address 127.10.71.190:37699)
0619 14:57:50.491610 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:50.491613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:50.491670 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:57:50.491795 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:50.491796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:50.491836 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:50.491892 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:50.492279 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.492887 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.493653 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.493858 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.493931 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:50.496438 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:46991: BlockingRecv error: recv got EOF from 127.10.71.130:46991 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:57:50 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:50 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:50.668916 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:50.663470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:56215)
0619 14:57:50.663737 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:50.663751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:50.663837 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:50.664097 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:50.664100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:50.664273 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:50.664531 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:50.664550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.665274 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.665277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:50.665839 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:50.665847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.666264 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.666267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:50.666326 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:50.666837 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:50.666851 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:50.668813 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":63,"threads_started":1}
W20260619 14:57:50.669026 19708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:50.663696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:56215 (local address 127.10.71.190:37699)
0619 14:57:50.663822 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:50.663825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:50.663837 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:50.663933 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:50.663935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:50.663976 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:50.664038 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:50.664647 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.665167 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.665967 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:50.666168 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:50.666245 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:50.668920 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:56215: BlockingRecv error: recv got EOF from 127.10.71.131:56215 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:57:51 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:51 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:51.333174 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:51.327741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:43283)
0619 14:57:51.328022 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:51.328045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:51.328134 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:51.328448 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:51.328450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:51.328619 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:51.328711 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:51.328715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.329633 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.329636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:51.330347 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:51.330356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.330759 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.330762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:51.330830 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:51.331273 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:51.331283 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:51.333074 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
W20260619 14:57:51.333293 19712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:51.327833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:43283 (local address 127.10.71.190:37699)
0619 14:57:51.328125 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:51.328129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:51.328141 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:57:51.328210 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:51.328212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:51.328251 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:51.328327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:51.328801 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.329516 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.330439 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.330668 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.330743 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:51.333174 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:43283: BlockingRecv error: recv got EOF from 127.10.71.129:43283 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
Jun 19 14:57:51 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:51 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:51.502580 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:51.497175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:33815)
0619 14:57:51.497551 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:51.497568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:51.497704 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:51.497973 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:51.497976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:51.498203 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:51.498341 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:51.498350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.499078 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.499084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:51.499618 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:51.499626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.500062 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.500066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:51.500147 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:51.500605 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:51.500617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:51.502407 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":85,"threads_started":1}
W20260619 14:57:51.502633 19712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:51.497218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:33815 (local address 127.10.71.190:37699)
0619 14:57:51.497389 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:51.497394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:51.497655 (+ 261us) server_negotiation.cc:407] Connection header received
0619 14:57:51.497808 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:51.497812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:51.497868 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:51.497962 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:51.498443 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.498960 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.499738 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.499950 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.500069 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:51.502522 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:33815: BlockingRecv error: recv got EOF from 127.10.71.130:33815 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 14:57:51 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:51 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:51.674932 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:51.669634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:48981)
0619 14:57:51.669878 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:51.669889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:51.670008 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:51.670253 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:51.670255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:51.670475 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:51.670577 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:51.670583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.671309 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.671314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:51.671903 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:51.671911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.672366 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.672369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:51.672449 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:51.672964 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:51.672975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:51.674805 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
W20260619 14:57:51.675051 19712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:51.669677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:48981 (local address 127.10.71.190:37699)
0619 14:57:51.669816 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:51.669821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:51.669953 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:57:51.670107 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:51.670109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:51.670157 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:51.670235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:51.670686 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.671203 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.672017 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:51.672236 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:51.672360 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:51.674919 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:48981: BlockingRecv error: recv got EOF from 127.10.71.131:48981 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 14:57:52 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:52 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:52.342757 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:52.333948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:42937)
0619 14:57:52.334200 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:52.334213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:52.334294 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:52.334516 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:52.334518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:52.334687 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:52.334814 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:52.334819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.335527 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.335530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:52.336084 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:52.336090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.336469 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.336472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:52.336532 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:52.340570 (+ 4038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:52.340581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:52.342653 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260619 14:57:52.342916 19716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:52.334002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:42937 (local address 127.10.71.190:37699)
0619 14:57:52.334262 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:52.334265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:52.334276 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:57:52.334374 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:52.334376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:52.334417 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:52.334485 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:52.334908 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.335442 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.336170 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.336384 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.336452 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:52.342790 (+ 6338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:42937: BlockingRecv error: recv got EOF from 127.10.71.129:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
Jun 19 14:57:52 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:52 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:52.508051 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:52.503347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:60781)
0619 14:57:52.503569 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:52.503581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:52.503664 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:52.503864 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:52.503866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:52.504017 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:52.504122 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:52.504127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.504791 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.504794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:52.505351 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:52.505358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.505749 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.505751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:52.505808 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:52.506252 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:52.506261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:52.507894 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
W20260619 14:57:52.508229 19716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:52.503423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:60781 (local address 127.10.71.190:37699)
0619 14:57:52.503545 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:52.503548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:52.503639 (+ 91us) server_negotiation.cc:407] Connection header received
0619 14:57:52.503733 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:52.503735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:52.503772 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:52.503837 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:52.504207 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.504669 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.505448 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.505660 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.505736 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:52.508093 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:60781: BlockingRecv error: recv got EOF from 127.10.71.130:60781 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 14:57:52 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:52 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:52.680763 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:52.675642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:47051)
0619 14:57:52.675905 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:52.675917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:52.676022 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:52.676228 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:52.676230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:52.676428 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:52.676516 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:52.676522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.677197 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.677200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:52.677729 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:52.677736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.678075 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.678078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:52.678132 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:52.678649 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:52.678660 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:52.680610 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
W20260619 14:57:52.680810 19716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:52.675679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:47051 (local address 127.10.71.190:37699)
0619 14:57:52.675815 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:52.675819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:52.675995 (+ 176us) server_negotiation.cc:407] Connection header received
0619 14:57:52.676093 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:52.676095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:52.676138 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:52.676198 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:52.676619 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.677112 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.677810 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:52.677995 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:52.678060 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:52.680714 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:47051: BlockingRecv error: recv got EOF from 127.10.71.131:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:57:53 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:53 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:53.348773 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:53.343547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:36363)
0619 14:57:53.343851 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:53.343864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:53.343954 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:53.344192 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:53.344194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:53.344476 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:53.344646 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:53.344653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.345384 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.345387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:53.345930 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:53.345937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.346365 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.346367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:53.346425 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:53.346849 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:53.346860 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:53.348598 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260619 14:57:53.348839 19720 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:53.343601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:36363 (local address 127.10.71.190:37699)
0619 14:57:53.343854 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:53.343861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:53.343920 (+ 59us) server_negotiation.cc:407] Connection header received
0619 14:57:53.344033 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:53.344035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:53.344081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:53.344148 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:53.344746 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.345292 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.346059 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.346276 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.346346 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:53.348722 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:36363: BlockingRecv error: recv got EOF from 127.10.71.129:36363 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
Jun 19 14:57:53 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:53 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:53.514020 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:53.508806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:60909)
0619 14:57:53.509100 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:53.509112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:53.509223 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:53.509475 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:53.509478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:53.509727 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:53.509818 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:53.509824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.510568 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.510572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:53.511083 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:53.511090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.511521 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.511525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:53.511615 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:53.512166 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:53.512178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:53.513870 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
W20260619 14:57:53.514099 19720 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:53.508877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:60909 (local address 127.10.71.190:37699)
0619 14:57:53.509024 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:53.509030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:53.509181 (+ 151us) server_negotiation.cc:407] Connection header received
0619 14:57:53.509314 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:53.509317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:53.509371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:53.509455 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:53.509950 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.510445 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.511207 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.511418 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.511520 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:53.513981 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:60909: BlockingRecv error: recv got EOF from 127.10.71.130:60909 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:57:53 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:53 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:53.686371 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:53.681460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:52255)
0619 14:57:53.681725 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:53.681735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:53.681844 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:53.682095 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:53.682098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:53.682279 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:53.682358 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:53.682363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.683057 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.683069 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:53.683583 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:53.683590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.683983 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.683986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:53.684069 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:53.684532 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:53.684541 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:53.686216 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
W20260619 14:57:53.686420 19720 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:53.681501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:52255 (local address 127.10.71.190:37699)
0619 14:57:53.681643 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:53.681648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:53.681798 (+ 150us) server_negotiation.cc:407] Connection header received
0619 14:57:53.681937 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:53.681941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:53.681991 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:53.682067 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:53.682466 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.682961 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.683691 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:53.683888 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:53.683981 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:53.686326 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:52255: BlockingRecv error: recv got EOF from 127.10.71.131:52255 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 14:57:54 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:54 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:54.355436 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:54.349698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:57697)
0619 14:57:54.349970 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:54.349984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:54.350082 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:54.350427 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:54.350430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:54.350676 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:54.350808 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:54.350819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.351748 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.351752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:54.352326 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:54.352335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.352793 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.352797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:54.352882 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:54.353435 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:54.353448 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:54.355271 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20260619 14:57:54.355513 19724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:54.349738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:57697 (local address 127.10.71.190:37699)
0619 14:57:54.350014 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:54.350019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:54.350035 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:57:54.350194 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:54.350198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:54.350261 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:54.350365 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:54.350956 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.351638 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.352446 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.352682 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.352790 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:54.355390 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:57697: BlockingRecv error: recv got EOF from 127.10.71.129:57697 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
Jun 19 14:57:54 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:54 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:54.520196 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:54.514711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:44705)
0619 14:57:54.514988 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:54.515003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:54.515100 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:54.515360 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:54.515362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:54.515546 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:54.515655 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:54.515661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.516387 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.516390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:54.516938 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:54.516945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.517349 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.517351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:54.517419 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:54.518053 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:54.518068 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:54.520029 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
W20260619 14:57:54.520280 19724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:54.514799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:44705 (local address 127.10.71.190:37699)
0619 14:57:54.514940 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:54.514943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:54.515075 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:57:54.515183 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:54.515185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:54.515236 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:54.515318 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:54.515757 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.516311 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.517030 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.517251 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.517323 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:54.520165 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:44705: BlockingRecv error: recv got EOF from 127.10.71.130:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:57:54 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:54 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:54.692896 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:54.687154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:51275)
0619 14:57:54.687395 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:54.687408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:54.687505 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:54.687895 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:54.687900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:54.688096 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:54.688193 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:54.688199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.688968 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.688970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:54.689517 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:54.689524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.689957 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.689959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:54.690019 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:54.690579 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:54.690600 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:54.692731 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
W20260619 14:57:54.692965 19724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:54.687181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:51275 (local address 127.10.71.190:37699)
0619 14:57:54.687338 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:54.687342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:54.687476 (+ 134us) server_negotiation.cc:407] Connection header received
0619 14:57:54.687661 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:54.687664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:54.687716 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:54.687821 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:54.688317 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.688866 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.689640 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:54.689859 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:54.689946 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:54.692856 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:51275: BlockingRecv error: recv got EOF from 127.10.71.131:51275 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 14:57:55 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:55 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:55.361081 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:55.356252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:44115)
0619 14:57:55.356573 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:55.356586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:55.356719 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:55.356938 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:55.356940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:55.357119 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:55.357227 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:55.357232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.357904 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.357907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:55.358414 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:55.358421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.358802 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.358804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:55.358860 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:55.359271 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:55.359281 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:55.360928 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":75,"threads_started":1}
W20260619 14:57:55.361129 19728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:55.356368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:44115 (local address 127.10.71.190:37699)
0619 14:57:55.356575 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:55.356579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:55.356670 (+ 91us) server_negotiation.cc:407] Connection header received
0619 14:57:55.356792 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:55.356794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:55.356832 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:55.356901 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:55.357313 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.357819 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.358509 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.358715 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.358784 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:55.361026 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:44115: BlockingRecv error: recv got EOF from 127.10.71.129:44115 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
Jun 19 14:57:55 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:55 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:55.525852 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:55.521032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:57923)
0619 14:57:55.521320 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:55.521333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:55.521449 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:55.521679 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:55.521680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:55.521845 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:55.521932 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:55.521937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.522632 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.522635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:55.523112 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:55.523119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.523507 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.523510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:55.523569 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:55.524092 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:55.524103 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:55.525746 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
W20260619 14:57:55.526077 19728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:55.521086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:57923 (local address 127.10.71.190:37699)
0619 14:57:55.521232 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:55.521236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:55.521385 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:57:55.521549 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:55.521552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:55.521605 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:55.521663 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:55.522022 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.522530 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.523228 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.523419 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.523485 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:55.525953 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:57923: BlockingRecv error: recv got EOF from 127.10.71.130:57923 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 14:57:55 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:55 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:55.698611 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:55.693551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:60553)
0619 14:57:55.693836 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:55.693850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:55.693952 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:55.694194 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:55.694196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:55.694382 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:55.694482 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:55.694487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.695188 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.695190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:55.695757 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:55.695765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.696186 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.696188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:55.696263 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:55.696744 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:55.696755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:55.698486 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":60,"threads_started":1}
W20260619 14:57:55.698719 19728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:55.693706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:60553 (local address 127.10.71.190:37699)
0619 14:57:55.693842 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:55.693846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:55.693919 (+ 73us) server_negotiation.cc:407] Connection header received
0619 14:57:55.694029 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:55.694031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:55.694075 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:55.694133 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:55.694585 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.695090 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.695865 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:55.696075 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:55.696151 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:55.698616 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:60553: BlockingRecv error: recv got EOF from 127.10.71.131:60553 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 14:57:56 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:56 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:56.367350 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:56.361858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:46599)
0619 14:57:56.362095 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:56.362108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:56.362187 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:56.362436 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:56.362438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:56.362607 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:56.362698 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:56.362703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.363420 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.363423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:56.364230 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:56.364242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.364841 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.364844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:56.364936 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:56.365442 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:56.365453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:56.367182 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260619 14:57:56.367444 19732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:56.361989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:46599 (local address 127.10.71.190:37699)
0619 14:57:56.362219 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:56.362222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:56.362247 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:57:56.362293 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:56.362295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:56.362333 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:56.362395 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:56.362796 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.363290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.364390 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.364738 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.364834 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:56.367307 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:46599: BlockingRecv error: recv got EOF from 127.10.71.129:46599 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
Jun 19 14:57:56 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:56 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:56.531394 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:56.526554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:56589)
0619 14:57:56.526783 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:56.526795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:56.526875 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:56.527068 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:56.527070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:56.527222 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:56.527307 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:56.527312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.528101 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.528103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:56.528603 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:56.528611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.528958 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.528960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:56.529010 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:56.529491 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:56.529501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:56.531286 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
W20260619 14:57:56.531512 19732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:56.526630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:56589 (local address 127.10.71.190:37699)
0619 14:57:56.526755 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:56.526763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:56.526838 (+ 75us) server_negotiation.cc:407] Connection header received
0619 14:57:56.526935 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:56.526937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:56.526978 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:56.527033 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:56.527394 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.527988 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.528689 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.528876 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.528942 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:56.531408 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:56589: BlockingRecv error: recv got EOF from 127.10.71.130:56589 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:57:56 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:56 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:56.704397 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:56.699359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:45199)
0619 14:57:56.699628 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:56.699640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:56.699729 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:56.699935 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:56.699937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:56.700115 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:56.700213 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:56.700218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.700848 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.700850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:56.701398 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:56.701406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.701887 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.701891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:56.701978 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:56.702493 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:56.702511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:56.704240 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
W20260619 14:57:56.704494 19732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:56.699432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:45199 (local address 127.10.71.190:37699)
0619 14:57:56.699541 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:56.699544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:56.699700 (+ 156us) server_negotiation.cc:407] Connection header received
0619 14:57:56.699797 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:56.699799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:56.699838 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:56.699896 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:56.700322 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.700766 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.701512 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:56.701771 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:56.701838 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:56.704398 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:45199: BlockingRecv error: recv got EOF from 127.10.71.131:45199 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 14:57:57 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:57 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:57.376446 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:57.368110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:44321)
0619 14:57:57.368427 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:57.368440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:57.368525 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:57.368773 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:57.368775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:57.368941 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:57.369033 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:57.369038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.369828 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.369831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:57.370402 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:57.370409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.370888 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.370891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:57.370957 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:57.374413 (+ 3456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:57.374424 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:57.376245 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
W20260619 14:57:57.376644 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:57.368206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:44321 (local address 127.10.71.190:37699)
0619 14:57:57.368483 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:57.368486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:57.368501 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:57:57.368615 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:57.368618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:57.368662 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:57.368742 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:57.369154 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.369715 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.370500 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.370751 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.370834 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:57.376444 (+ 5610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:44321: BlockingRecv error: recv got EOF from 127.10.71.129:44321 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
Jun 19 14:57:57 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:57 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:57.537237 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:57.532106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:39837)
0619 14:57:57.532378 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:57.532390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:57.532499 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:57.532764 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:57.532766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:57.532926 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:57.533018 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:57.533024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.533727 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.533730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:57.534206 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:57.534214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.534611 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.534613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:57.534670 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:57.535255 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:57.535267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:57.537124 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
W20260619 14:57:57.537349 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:57.532119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:39837 (local address 127.10.71.190:37699)
0619 14:57:57.532292 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:57.532295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:57.532470 (+ 175us) server_negotiation.cc:407] Connection header received
0619 14:57:57.532599 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:57.532601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:57.532655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:57.532715 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:57.533113 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.533598 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.534328 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.534518 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.534596 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:57.537230 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:39837: BlockingRecv error: recv got EOF from 127.10.71.130:39837 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 14:57:57 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:57 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:57.709988 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:57.705143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:53605)
0619 14:57:57.705416 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:57.705429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:57.705525 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:57.705760 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:57.705762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:57.705923 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:57.706017 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:57.706022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.706679 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.706681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:57.707154 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:57.707160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.707487 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.707489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:57.707541 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:57.708014 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:57.708024 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:57.709889 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
W20260619 14:57:57.710244 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:57.705217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:53605 (local address 127.10.71.190:37699)
0619 14:57:57.705358 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:57.705363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:57.705495 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:57:57.705614 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:57.705616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:57.705659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:57.705722 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:57.706111 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.706603 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.707234 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:57.707411 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:57.707471 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:57.709986 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:53605: BlockingRecv error: recv got EOF from 127.10.71.131:53605 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:57:58 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:58 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:58.381796 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:58.377151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:33247)
0619 14:57:58.377393 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:58.377404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:58.377483 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:58.377697 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:58.377699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:58.377871 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:58.377961 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:58.377965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.378601 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.378603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:58.379087 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:58.379093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.379442 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.379444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:58.379494 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:58.379979 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:58.379990 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:58.381629 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260619 14:57:58.381923 19740 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:58.377206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:33247 (local address 127.10.71.190:37699)
0619 14:57:58.377411 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:58.377414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:58.377447 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:57:58.377550 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:58.377552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:58.377588 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:58.377664 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:58.378053 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.378522 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.379178 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.379360 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.379424 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:58.381795 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:33247: BlockingRecv error: recv got EOF from 127.10.71.129:33247 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
Jun 19 14:57:58 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:58 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:58.542889 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:58.537850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:34719)
0619 14:57:58.538092 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:58.538103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:58.538182 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:58.538371 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:58.538373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:58.538513 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:58.538595 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:58.538600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.539275 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.539278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:58.539895 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:58.539904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.540432 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.540435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:58.540496 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:58.541046 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:58.541059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:58.542747 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":56,"threads_started":1}
W20260619 14:57:58.542980 19740 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:58.537949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:34719 (local address 127.10.71.190:37699)
0619 14:57:58.538094 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:58.538097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:58.538148 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:57:58.538246 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:58.538248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:58.538283 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:58.538336 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:58.538681 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.539151 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.539989 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.540312 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.540406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:58.542863 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:34719: BlockingRecv error: recv got EOF from 127.10.71.130:34719 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:57:58 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:58 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:58.715318 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:58.710614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:59103)
0619 14:57:58.710844 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:58.710855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:58.710932 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:58.711121 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:58.711122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:58.711268 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:58.711370 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:58.711375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.712212 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.712215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:58.712770 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:58.712778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.713158 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.713160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:58.713211 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:58.713623 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:58.713640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:58.715187 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
W20260619 14:57:58.715421 19740 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:58.710724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:59103 (local address 127.10.71.190:37699)
0619 14:57:58.710839 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:58.710846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:58.710896 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:57:58.710996 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:58.710998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:58.711035 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:58.711090 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:58.711469 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.712112 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.712884 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:58.713077 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:58.713142 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:58.715315 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:59103: BlockingRecv error: recv got EOF from 127.10.71.131:59103 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:57:59 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:59 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:59.387616 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:59.382669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:54877)
0619 14:57:59.382934 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:59.382947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:59.383036 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:59.383252 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:59.383254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:59.383413 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:59.383509 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:59.383513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.384198 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.384200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:59.384757 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:59.384766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.385150 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.385152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:59.385207 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:59.385710 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:59.385721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:59.387446 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
W20260619 14:57:59.387768 19744 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:59.382704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:54877 (local address 127.10.71.190:37699)
0619 14:57:59.382948 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:59.382951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:59.382989 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:57:59.383112 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:59.383114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:59.383154 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:59.383221 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:59.383586 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.384118 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.384841 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.385039 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.385127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:59.387567 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:54877: BlockingRecv error: recv got EOF from 127.10.71.129:54877 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
Jun 19 14:57:59 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:59 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:59.549197 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:59.543560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:40347)
0619 14:57:59.543837 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:59.543857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:59.543975 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:59.544229 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:59.544232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:59.544502 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:59.544601 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:59.544607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.545285 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.545289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:59.545872 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:59.545879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.546338 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.546341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:59.546428 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:59.546914 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:59.546925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:59.549085 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
W20260619 14:57:59.549314 19744 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:59.543631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:40347 (local address 127.10.71.190:37699)
0619 14:57:59.543763 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:59.543767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:59.543923 (+ 156us) server_negotiation.cc:407] Connection header received
0619 14:57:59.544083 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:59.544090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:59.544132 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:59.544210 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:59.544710 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.545187 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.545995 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.546228 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.546337 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:59.549187 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:40347: BlockingRecv error: recv got EOF from 127.10.71.130:40347 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:57:59 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:57:59 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:57:59.722033 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:57:59.716033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:40761)
0619 14:57:59.716381 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 14:57:59.716393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:57:59.716478 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:57:59.716769 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:57:59.716771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:57:59.716913 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:57:59.716999 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:59.717003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.717871 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.717877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:59.718943 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:57:59.718958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.719398 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.719403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:57:59.719520 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:59.719980 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:57:59.719997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:57:59.721864 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
W20260619 14:57:59.722191 19744 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:57:59.716158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:40761 (local address 127.10.71.190:37699)
0619 14:57:59.716331 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:57:59.716338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:57:59.716460 (+ 122us) server_negotiation.cc:407] Connection header received
0619 14:57:59.716575 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:57:59.716580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:57:59.716657 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:57:59.716741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:57:59.717109 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.717757 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.719089 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:57:59.719285 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:57:59.719360 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:57:59.722035 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:40761: BlockingRecv error: recv got EOF from 127.10.71.131:40761 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:58:00 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:00 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:00.393543 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:00.388404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:52627)
0619 14:58:00.388673 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:00.388686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:00.388778 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:00.389011 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:00.389013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:00.389199 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:00.389293 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:00.389297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.390009 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.390011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:00.390569 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:00.390577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.390964 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.390966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:00.391026 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:00.391523 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:00.391534 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:00.393380 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
W20260619 14:58:00.393611 19748 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:00.388436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:52627 (local address 127.10.71.190:37699)
0619 14:58:00.388673 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:00.388676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:00.388750 (+ 74us) server_negotiation.cc:407] Connection header received
0619 14:58:00.388856 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:00.388859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:00.388900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:00.388976 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:00.389390 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.389916 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.390665 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.390872 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.390946 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:00.393491 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:52627: BlockingRecv error: recv got EOF from 127.10.71.129:52627 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
Jun 19 14:58:00 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:00 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:00.555404 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:00.549827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:46745)
0619 14:58:00.550093 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:00.550106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:00.550200 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:00.550438 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:00.550441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:00.550649 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:00.550758 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:00.550764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.551517 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.551523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:00.552154 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:00.552164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.552786 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.552792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:00.552885 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:00.553486 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:00.553503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:00.555276 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
W20260619 14:58:00.555588 19748 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:00.549906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:46745 (local address 127.10.71.190:37699)
0619 14:58:00.550075 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:00.550079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:00.550170 (+ 91us) server_negotiation.cc:407] Connection header received
0619 14:58:00.550275 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:00.550277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:00.550319 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:00.550412 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:00.550872 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.551410 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.552277 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.552642 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.552769 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:00.555420 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:46745: BlockingRecv error: recv got EOF from 127.10.71.130:46745 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:58:00 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:00 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:00.727705 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:00.722641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:47327)
0619 14:58:00.722907 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:00.722918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:00.723034 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:00.723288 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:00.723291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:00.723473 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:00.723557 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:00.723562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.724251 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.724255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:00.724816 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:00.724824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.725262 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.725265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:00.725348 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:00.725823 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:00.725834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:00.727598 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
W20260619 14:58:00.727875 19748 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:00.722709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:47327 (local address 127.10.71.190:37699)
0619 14:58:00.722831 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:00.722835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:00.722984 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:58:00.723132 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:00.723134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:00.723184 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:00.723261 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:00.723666 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.724150 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.724931 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:00.725155 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:00.725262 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:00.727718 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:47327: BlockingRecv error: recv got EOF from 127.10.71.131:47327 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 14:58:01 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:01 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:01.399729 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:01.394452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:35889)
0619 14:58:01.394701 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:01.394713 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:01.394797 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:01.394997 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:01.394999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:01.395176 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:01.395264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:01.395269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.396002 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.396006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:01.396641 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:01.396651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.397296 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.397300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:01.397405 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:01.397951 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:01.397964 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:01.399621 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
W20260619 14:58:01.399873 19752 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:01.394532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:35889 (local address 127.10.71.190:37699)
0619 14:58:01.394775 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:01.394778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:01.394789 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:58:01.394859 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:01.394861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:01.394896 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:01.394965 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:01.395358 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.395885 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.396831 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.397176 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.397274 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:01.399751 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:35889: BlockingRecv error: recv got EOF from 127.10.71.129:35889 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":106,"threads_started":1}
Jun 19 14:58:01 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:01 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:01.560945 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:01.556328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:44913)
0619 14:58:01.556554 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:01.556567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:01.556653 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:01.556849 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:01.556851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:01.556989 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:01.557082 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:01.557087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.557818 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.557820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:01.558348 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:01.558355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.558714 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.558716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:01.558770 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:01.559196 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:01.559206 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:01.560850 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":58,"threads_started":1}
W20260619 14:58:01.561110 19752 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:01.556432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:44913 (local address 127.10.71.190:37699)
0619 14:58:01.556572 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:01.556575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:01.556611 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:58:01.556747 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:01.556748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:01.556784 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:01.556838 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:01.557219 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.557731 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.558450 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.558632 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.558698 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:01.560979 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:44913: BlockingRecv error: recv got EOF from 127.10.71.130:44913 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:58:01 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:01 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:01.733084 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:01.728407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:53641)
0619 14:58:01.728643 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:01.728654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:01.728740 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:01.728932 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:01.728934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:01.729093 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:01.729180 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:01.729185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.729825 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.729827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:01.730292 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:01.730299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.730652 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.730655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:01.730709 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:01.731177 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:01.731187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:01.732965 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
W20260619 14:58:01.733208 19752 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:01.728441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:53641 (local address 127.10.71.190:37699)
0619 14:58:01.728554 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:01.728562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:01.728710 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:58:01.728797 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:01.728799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:01.728839 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:01.728901 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:01.729265 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.729749 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.730377 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:01.730568 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:01.730635 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:01.733103 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:53641: BlockingRecv error: recv got EOF from 127.10.71.131:53641 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 14:58:02 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:02 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:02.410218 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:02.400492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:57409)
0619 14:58:02.400763 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:02.400777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:02.400870 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:02.401157 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:02.401160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:02.401383 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:02.401487 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:02.401493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.402159 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.402161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:02.402709 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:02.402716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.403131 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.403133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:02.403183 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:02.408521 (+ 5338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:02.408532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:02.410112 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
W20260619 14:58:02.410351 19756 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:02.400542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:57409 (local address 127.10.71.190:37699)
0619 14:58:02.400858 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:02.400861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:02.400875 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:58:02.400970 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:02.400973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:02.401032 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:02.401120 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:02.401599 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.402074 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.402801 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.403024 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.403119 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:02.410236 (+ 7117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:57409: BlockingRecv error: recv got EOF from 127.10.71.129:57409 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":74,"threads_started":1}
Jun 19 14:58:02 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:02 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:02.566968 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:02.561670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:48011)
0619 14:58:02.561941 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:02.561960 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:02.562080 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:02.562340 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:02.562343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:02.562523 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:02.562602 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:02.562607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.563304 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.563309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:02.563844 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:02.563851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.564340 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.564343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:02.564428 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:02.564977 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:02.564989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:02.566853 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
W20260619 14:58:02.567126 19756 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:02.561742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:48011 (local address 127.10.71.190:37699)
0619 14:58:02.561904 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:02.561909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:02.562026 (+ 117us) server_negotiation.cc:407] Connection header received
0619 14:58:02.562194 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:02.562197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:02.562239 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:02.562319 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:02.562703 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.563205 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.563958 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.564207 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.564328 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:02.566978 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:48011: BlockingRecv error: recv got EOF from 127.10.71.130:48011 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 14:58:02 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:02 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.131@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:02.739091 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:02.733847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.131:40921)
0619 14:58:02.734134 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:02.734146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:02.734237 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:02.734463 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:02.734465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:02.734628 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:02.734712 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:02.734717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.735358 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.735359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:02.735893 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:02.735900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.736447 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.736449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:02.736511 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:02.737045 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:02.737056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:02.738938 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
W20260619 14:58:02.739173 19756 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:02.733962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:40921 (local address 127.10.71.190:37699)
0619 14:58:02.734120 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:02.734123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:02.734212 (+ 89us) server_negotiation.cc:407] Connection header received
0619 14:58:02.734324 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:02.734326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:02.734369 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:02.734426 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:02.734800 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.735270 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.736013 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:02.736261 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:02.736430 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:02.739041 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.131:40921: BlockingRecv error: recv got EOF from 127.10.71.131:40921 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 14:58:03 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:03 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.129@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:03.415956 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:03.411090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.129:54525)
0619 14:58:03.411325 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:03.411336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:03.411407 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:03.411625 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:03.411626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:03.411775 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:03.411857 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:03.411861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.412687 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.412691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:03.413294 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:03.413316 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.413707 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.413709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:03.413763 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:03.414192 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:03.414202 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:03.415818 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
W20260619 14:58:03.416110 19760 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:03.411131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:54525 (local address 127.10.71.190:37699)
0619 14:58:03.411364 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:03.411367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:03.411378 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:58:03.411483 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:03.411485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:03.411521 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:03.411596 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:03.411937 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.412585 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.413426 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.413622 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.413690 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:03.415969 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.129:54525: BlockingRecv error: recv got EOF from 127.10.71.129:54525 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
Jun 19 14:58:03 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:58:03 dist-test-slave-3qdh krb5kdc[18702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.71.130@KRBTEST.COM for oryx/127.10.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:58:03.572926 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:03.567766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37699 (local address 127.10.71.130:57479)
0619 14:58:03.568057 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:03.568072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:03.568159 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:03.568370 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:03.568372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:03.568529 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:03.568628 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:03.568633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.569346 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.569348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:03.569915 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:03.569925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.570454 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.570457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:03.570538 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:03.571109 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:03.571119 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:03.572784 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.71.190:37699: Server oryx/127.10.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
W20260619 14:58:03.572975 19760 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:03.567849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:57479 (local address 127.10.71.190:37699)
0619 14:58:03.567997 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:03.568004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:03.568141 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:58:03.568223 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:03.568225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:03.568263 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:03.568350 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:03.568721 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.569254 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.570025 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.570344 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.570438 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:03.572883 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.71.130:57479: BlockingRecv error: recv got EOF from 127.10.71.130:57479 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:58:03.648159 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19285
I20260619 14:58:03.652904 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19437
I20260619 14:58:03.657830 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19569
I20260619 14:58:03.662315 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19215
2026-06-19T14:58:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17597 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:58:03Z Disabled control of system clock
I20260619 14:58:03.690874 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:35257
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:58:03.790135 19765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:03.790323 19765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:03.790350 19765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:03.793982 19765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:58:03.794026 19765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:03.794041 19765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:03.794056 19765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:58:03.794071 19765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:58:03.798580 19765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:35257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:03.799401 19765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:03.800552 19765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:03.807638 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:03.807644 19770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:03.807761 19765 server_base.cc:1034] running on GCE node
W20260619 14:58:03.807654 19771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:03.808503 19765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:03.809432 19765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:03.810595 19765 hybrid_clock.cc:648] HybridClock initialized: now 1781881083810574 us; error 34 us; skew 500 ppm
I20260619 14:58:03.812150 19765 webserver.cc:460] Webserver started at http://127.10.71.190:44177/ using document root <none> and password file <none>
I20260619 14:58:03.812722 19765 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:03.812804 19765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:03.812983 19765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:03.814785 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "93b729b565e04953b809eaa3c054d1ec"
format_stamp: "Formatted at 2026-06-19 14:58:03 on dist-test-slave-3qdh"
server_key: "1fa6c15b63a7231a70b56c7b29f577e9"
server_key_iv: "07339270f73279da06ea68c421fb2338"
server_key_version: "encryptionkey@0"
I20260619 14:58:03.815267 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "93b729b565e04953b809eaa3c054d1ec"
format_stamp: "Formatted at 2026-06-19 14:58:03 on dist-test-slave-3qdh"
server_key: "1fa6c15b63a7231a70b56c7b29f577e9"
server_key_iv: "07339270f73279da06ea68c421fb2338"
server_key_version: "encryptionkey@0"
I20260619 14:58:03.818392 19765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:58:03.820541 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:03.821548 19765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:58:03.821669 19765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "93b729b565e04953b809eaa3c054d1ec"
format_stamp: "Formatted at 2026-06-19 14:58:03 on dist-test-slave-3qdh"
server_key: "1fa6c15b63a7231a70b56c7b29f577e9"
server_key_iv: "07339270f73279da06ea68c421fb2338"
server_key_version: "encryptionkey@0"
I20260619 14:58:03.821759 19765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:03.845882 19765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:03.846416 19765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:03.846589 19765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:03.852392 19765 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:35257
I20260619 14:58:03.852417 19831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:35257 every 8 connection(s)
I20260619 14:58:03.853137 19765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:58:03.855181 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19765
I20260619 14:58:03.855283 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:58:03.855543 10526 external_mini_cluster.cc:1428] Setting key 358ceb71498d09305a9f465103df5dc3
I20260619 14:58:03.855990 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:03.861770 19832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec: Bootstrap starting.
I20260619 14:58:03.863713 19832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:03.863739 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:03.856860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.10.71.190:35257)
0619 14:58:03.857395 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:03.857408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:03.857453 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:58:03.858289 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:03.858304 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:03.858614 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:03.858846 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:03.859164 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.859982 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.861122 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:03.861508 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:03.861609 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:03.862312 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:03.862316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:03.862325 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:03.862653 (+ 328us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:03.862698 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:03.862725 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:03.862788 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:03.862951 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:58:03.863196 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260619 14:58:03.864416 19832 log.cc:824] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:03.865900 19832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec: No bootstrap required, opened a new log
I20260619 14:58:03.868799 19832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b729b565e04953b809eaa3c054d1ec" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35257 } }
I20260619 14:58:03.868973 19832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:03.869046 19832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b729b565e04953b809eaa3c054d1ec, State: Initialized, Role: FOLLOWER
I20260619 14:58:03.869422 19832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b729b565e04953b809eaa3c054d1ec" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35257 } }
I20260619 14:58:03.869539 19832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:58:03.869592 19832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:58:03.869692 19832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:03.870364 19832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b729b565e04953b809eaa3c054d1ec" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35257 } }
I20260619 14:58:03.870631 19832 leader_election.cc:302] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b729b565e04953b809eaa3c054d1ec; no voters:
I20260619 14:58:03.870858 19832 leader_election.cc:288] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:58:03.870950 19837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:58:03.871177 19837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [term 1 LEADER]: Becoming Leader. State: Replica: 93b729b565e04953b809eaa3c054d1ec, State: Running, Role: LEADER
I20260619 14:58:03.871460 19837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b729b565e04953b809eaa3c054d1ec" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35257 } }
I20260619 14:58:03.871791 19832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:58:03.872826 19839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b729b565e04953b809eaa3c054d1ec. Latest consensus state: current_term: 1 leader_uuid: "93b729b565e04953b809eaa3c054d1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b729b565e04953b809eaa3c054d1ec" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35257 } } }
I20260619 14:58:03.872934 19839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:03.873224 19846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:58:03.873239 19838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b729b565e04953b809eaa3c054d1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b729b565e04953b809eaa3c054d1ec" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 35257 } } }
I20260619 14:58:03.873428 19838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:03.876080 19846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:58:03.881002 19846 catalog_manager.cc:1347] Generated new cluster ID: 424192a324774ddd8de333056ab2a764
I20260619 14:58:03.881089 19846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:58:03.894091 19846 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:58:03.895038 19846 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:58:03.900061 19846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b729b565e04953b809eaa3c054d1ec: Generated new TSK 0
I20260619 14:58:03.900705 19846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:58:03.905467 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35257
--builtin_ntp_servers=127.10.71.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:58:04.002449 19856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:04.002640 19856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:04.002669 19856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:04.006016 19856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:04.006059 19856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:04.006108 19856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:58:04.010119 19856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:04.010800 19856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:04.011816 19856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:04.019711 19862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.019706 19861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.019768 19856 server_base.cc:1034] running on GCE node
W20260619 14:58:04.019713 19864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.020291 19856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:04.020815 19856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:04.021968 19856 hybrid_clock.cc:648] HybridClock initialized: now 1781881084021954 us; error 26 us; skew 500 ppm
I20260619 14:58:04.023537 19856 webserver.cc:460] Webserver started at http://127.10.71.129:34243/ using document root <none> and password file <none>
I20260619 14:58:04.023998 19856 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:04.024075 19856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:04.024255 19856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:04.025982 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "c69e7a540a8f4c12ae674775f2a257ea"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "031f0fce45b1a572d2b699cd8d2f2484"
server_key_iv: "25820d7815282652e53862ee3a35d46c"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.026503 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "c69e7a540a8f4c12ae674775f2a257ea"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "031f0fce45b1a572d2b699cd8d2f2484"
server_key_iv: "25820d7815282652e53862ee3a35d46c"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.029596 19856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:58:04.031582 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.032656 19856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:58:04.032799 19856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "c69e7a540a8f4c12ae674775f2a257ea"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "031f0fce45b1a572d2b699cd8d2f2484"
server_key_iv: "25820d7815282652e53862ee3a35d46c"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.032893 19856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:04.048188 19856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:04.048714 19856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:04.048877 19856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:04.049352 19856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:04.050091 19856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:04.050163 19856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.050230 19856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:04.050274 19856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.057732 19856 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:38539
I20260619 14:58:04.057811 19983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:38539 every 8 connection(s)
I20260619 14:58:04.058419 19856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:58:04.060596 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19856
I20260619 14:58:04.060717 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:58:04.061038 10526 external_mini_cluster.cc:1428] Setting key 293525e46f9b8f58f89cb3e7a7050eae
I20260619 14:58:04.062732 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35257
--builtin_ntp_servers=127.10.71.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:58:04.066507 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.059907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:38301 (local address 127.10.71.190:35257)
0619 14:58:04.060021 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.060029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.060967 (+ 938us) server_negotiation.cc:407] Connection header received
0619 14:58:04.062172 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.062175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.062227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.062324 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.063142 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.063640 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.064777 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.064977 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.065078 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:04.066018 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:04.066021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:04.066024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:04.066043 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:04.066063 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:04.066070 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:04.066146 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:04.066302 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:58:04.066374 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:58:04.067167 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.060140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35257 (local address 127.10.71.129:38301)
0619 14:58:04.060817 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:04.060861 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:04.061987 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:04.062474 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:04.062480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:04.062545 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:04.063019 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:04.063031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.063766 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.063769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:04.064652 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:04.064677 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.065111 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.065115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:04.065251 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:04.065770 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:58:04.065772 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:04.065805 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:58:04.065859 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:04.066160 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:04.066170 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:04.066180 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:58:04.066288 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:58:04.066521 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":168,"threads_started":1}
I20260619 14:58:04.068185 19984 heartbeater.cc:347] Connected to a master server at 127.10.71.190:35257
I20260619 14:58:04.068408 19984 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:04.068805 19984 heartbeater.cc:511] Master 127.10.71.190:35257 requested a full tablet report, sending...
I20260619 14:58:04.070243 19796 ts_manager.cc:194] Registered new tserver with Master: c69e7a540a8f4c12ae674775f2a257ea (127.10.71.129:38539)
I20260619 14:58:04.071229 19796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.129:38301
W20260619 14:58:04.159389 19987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:04.159636 19987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:04.159723 19987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:04.163230 19987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:04.163302 19987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:04.163381 19987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:58:04.167603 19987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:04.168465 19987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:04.169608 19987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:04.177445 19993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.177451 19992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.177510 19995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.177738 19987 server_base.cc:1034] running on GCE node
I20260619 14:58:04.178076 19987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:04.178550 19987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:04.179711 19987 hybrid_clock.cc:648] HybridClock initialized: now 1781881084179707 us; error 29 us; skew 500 ppm
I20260619 14:58:04.181262 19987 webserver.cc:460] Webserver started at http://127.10.71.130:35007/ using document root <none> and password file <none>
I20260619 14:58:04.181720 19987 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:04.181806 19987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:04.182000 19987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:04.183773 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "977298a507de477da1db271875b92c7c"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "67094bbb547630ddb4f932dd8efdfa45"
server_key_iv: "8d05655d3fff246c5c2b857475509018"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.184298 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "977298a507de477da1db271875b92c7c"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "67094bbb547630ddb4f932dd8efdfa45"
server_key_iv: "8d05655d3fff246c5c2b857475509018"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.187367 19987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:58:04.189361 20001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.190271 19987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:58:04.190382 19987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "977298a507de477da1db271875b92c7c"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "67094bbb547630ddb4f932dd8efdfa45"
server_key_iv: "8d05655d3fff246c5c2b857475509018"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.190471 19987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:04.202379 19987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:04.202873 19987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:04.203032 19987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:04.203473 19987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:04.204164 19987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:04.204226 19987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.204288 19987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:04.204355 19987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.211737 19987 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:38187
I20260619 14:58:04.211796 20114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:38187 every 8 connection(s)
I20260619 14:58:04.212514 19987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:58:04.217821 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 19987
I20260619 14:58:04.217921 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:58:04.218187 10526 external_mini_cluster.cc:1428] Setting key 4d2361917e5c1af79ed318f7a4d7d06f
I20260619 14:58:04.220280 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35257
--builtin_ntp_servers=127.10.71.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:58:04.220744 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.214001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:53083 (local address 127.10.71.190:35257)
0619 14:58:04.214141 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.214144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.214960 (+ 816us) server_negotiation.cc:407] Connection header received
0619 14:58:04.216056 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.216059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.216107 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.216184 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.216992 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.217474 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.218840 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.219103 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.219210 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:04.220261 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:04.220265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:04.220266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:04.220277 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:04.220294 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:04.220339 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:04.220432 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:04.220601 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:58:04.220647 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:58:04.221668 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.214313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35257 (local address 127.10.71.130:53083)
0619 14:58:04.214832 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:04.214862 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:04.215880 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:04.216369 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:04.216376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:04.216449 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:04.216844 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:04.216855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.217576 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.217578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:04.218708 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:04.218717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.219287 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.219292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:04.219472 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:04.219967 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:58:04.219969 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:04.220011 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:58:04.220079 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:04.220459 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:04.220470 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:04.220478 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:58:04.220578 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:58:04.220810 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
I20260619 14:58:04.222786 20115 heartbeater.cc:347] Connected to a master server at 127.10.71.190:35257
I20260619 14:58:04.222983 20115 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:04.223356 20115 heartbeater.cc:511] Master 127.10.71.190:35257 requested a full tablet report, sending...
I20260619 14:58:04.224403 19796 ts_manager.cc:194] Registered new tserver with Master: 977298a507de477da1db271875b92c7c (127.10.71.130:38187)
I20260619 14:58:04.225349 19796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.130:53083
W20260619 14:58:04.316848 20118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:04.317072 20118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:04.317106 20118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:04.320650 20118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:04.320684 20118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:04.320729 20118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:58:04.325071 20118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:04.325882 20118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:04.326962 20118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:04.335008 20123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.335078 20124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.335103 20126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.335331 20118 server_base.cc:1034] running on GCE node
I20260619 14:58:04.335774 20118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:04.336422 20118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:04.337580 20118 hybrid_clock.cc:648] HybridClock initialized: now 1781881084337566 us; error 31 us; skew 500 ppm
I20260619 14:58:04.339171 20118 webserver.cc:460] Webserver started at http://127.10.71.131:34979/ using document root <none> and password file <none>
I20260619 14:58:04.339654 20118 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:04.339705 20118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:04.339900 20118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:04.341586 20118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "56653b4a880a46f3a38cae8300fe39ea"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "22c33a4b3ef048265e3762ece64e02f7"
server_key_iv: "1d598249db5f719787cd278e62503000"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.342046 20118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "56653b4a880a46f3a38cae8300fe39ea"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "22c33a4b3ef048265e3762ece64e02f7"
server_key_iv: "1d598249db5f719787cd278e62503000"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.345124 20118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:58:04.347400 20132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.348541 20118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:58:04.348675 20118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "56653b4a880a46f3a38cae8300fe39ea"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "22c33a4b3ef048265e3762ece64e02f7"
server_key_iv: "1d598249db5f719787cd278e62503000"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.348779 20118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:04.360848 20118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:04.361332 20118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:04.361486 20118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:04.361934 20118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:04.362664 20118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:04.362709 20118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.362766 20118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:04.362799 20118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.370658 20118 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:36799
I20260619 14:58:04.370682 20245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:36799 every 8 connection(s)
I20260619 14:58:04.371333 20118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:58:04.376391 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20118
I20260619 14:58:04.376500 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:58:04.376783 10526 external_mini_cluster.cc:1428] Setting key 08e9106114da620c741d48c6cc6428dd
I20260619 14:58:04.379624 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.373038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:53643 (local address 127.10.71.190:35257)
0619 14:58:04.373210 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.373218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.374110 (+ 892us) server_negotiation.cc:407] Connection header received
0619 14:58:04.375323 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.375326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.375380 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.375452 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.376237 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.376965 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.378127 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.378326 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.378398 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:04.379324 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:04.379327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:04.379329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:04.379339 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:04.379355 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:04.379361 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:04.379413 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:04.379486 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:58:04.379527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:58:04.380290 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.373432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:35257 (local address 127.10.71.131:53643)
0619 14:58:04.373974 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:04.374004 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:04.375135 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:04.375596 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:04.375602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:04.375671 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:04.376100 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:04.376111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.377099 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.377103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:04.377985 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:04.377994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:04.378451 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:04.378454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:04.378621 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:04.379134 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:58:04.379136 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:04.379165 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:58:04.379214 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:04.379412 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:04.379416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:04.379418 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 14:58:04.379486 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:58:04.379652 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260619 14:58:04.381253 20246 heartbeater.cc:347] Connected to a master server at 127.10.71.190:35257
I20260619 14:58:04.381479 20246 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:04.381880 20246 heartbeater.cc:511] Master 127.10.71.190:35257 requested a full tablet report, sending...
I20260619 14:58:04.382730 19796 ts_manager.cc:194] Registered new tserver with Master: 56653b4a880a46f3a38cae8300fe39ea (127.10.71.131:36799)
I20260619 14:58:04.383235 19796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.131:53643
I20260619 14:58:04.390175 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:58:04.392864 19835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:04.391878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.10.71.190:35257)
0619 14:58:04.392003 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.392007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.392216 (+ 209us) server_negotiation.cc:407] Connection header received
0619 14:58:04.392388 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.392391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.392447 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.392540 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.392761 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45542: BlockingRecv error: recv got EOF from 127.0.0.1:45542 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:58:04.393242 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19856
I20260619 14:58:04.398305 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19987
I20260619 14:58:04.403226 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20118
I20260619 14:58:04.408059 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 19765
2026-06-19T14:58:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (744 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:58:04Z Disabled control of system clock
I20260619 14:58:04.435921 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40867
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:58:04.536872 20257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:04.537098 20257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:04.537132 20257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:04.540994 20257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:58:04.541038 20257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:04.541056 20257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:04.541072 20257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:58:04.541119 20257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:58:04.545487 20257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:40867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:04.546391 20257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:04.547639 20257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:04.555321 20263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.555313 20265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.555313 20262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.555358 20257 server_base.cc:1034] running on GCE node
I20260619 14:58:04.556035 20257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:04.557061 20257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:04.558248 20257 hybrid_clock.cc:648] HybridClock initialized: now 1781881084558232 us; error 28 us; skew 500 ppm
I20260619 14:58:04.560066 20257 webserver.cc:460] Webserver started at http://127.10.71.190:34475/ using document root <none> and password file <none>
I20260619 14:58:04.560673 20257 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:04.560773 20257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:04.561026 20257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:04.562865 20257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "8bfa8c02078644e3abe1e81539fe3d83"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "409628cd61b964645fc85a77b53423ca"
server_key_iv: "c83c79160ed17dbe583c8a1e63f814eb"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.563387 20257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "8bfa8c02078644e3abe1e81539fe3d83"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "409628cd61b964645fc85a77b53423ca"
server_key_iv: "c83c79160ed17dbe583c8a1e63f814eb"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.566737 20257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:58:04.569096 20271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.570240 20257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:58:04.570376 20257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "8bfa8c02078644e3abe1e81539fe3d83"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "409628cd61b964645fc85a77b53423ca"
server_key_iv: "c83c79160ed17dbe583c8a1e63f814eb"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.570479 20257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:04.574692 20257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:04.575219 20257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:04.575390 20257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:04.581671 20257 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:40867
I20260619 14:58:04.581681 20323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:40867 every 8 connection(s)
I20260619 14:58:04.582479 20257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:58:04.585502 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:04.590498 20324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83: Bootstrap starting.
I20260619 14:58:04.590677 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20257
I20260619 14:58:04.590780 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:58:04.591053 10526 external_mini_cluster.cc:1428] Setting key 6abc02e74b934e4e75e2705d9f1e09e0
I20260619 14:58:04.592979 20324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:04.593670 20324 log.cc:824] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:04.595564 20324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83: No bootstrap required, opened a new log
I20260619 14:58:04.596251 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.592379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.10.71.190:40867)
0619 14:58:04.592949 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.592962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.592992 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:58:04.593618 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.593633 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.594007 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.594307 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.595080 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:04.595085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:04.595095 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:04.595105 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:04.595157 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:04.595173 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:04.595284 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:04.595376 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:58:04.595593 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260619 14:58:04.599038 20324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfa8c02078644e3abe1e81539fe3d83" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40867 } }
I20260619 14:58:04.599246 20324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:04.599323 20324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfa8c02078644e3abe1e81539fe3d83, State: Initialized, Role: FOLLOWER
I20260619 14:58:04.599715 20324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfa8c02078644e3abe1e81539fe3d83" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40867 } }
I20260619 14:58:04.599849 20324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:58:04.599928 20324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:58:04.600050 20324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:04.600859 20324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfa8c02078644e3abe1e81539fe3d83" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40867 } }
I20260619 14:58:04.601164 20324 leader_election.cc:302] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bfa8c02078644e3abe1e81539fe3d83; no voters:
I20260619 14:58:04.601421 20324 leader_election.cc:288] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:58:04.601555 20329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:58:04.601765 20329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [term 1 LEADER]: Becoming Leader. State: Replica: 8bfa8c02078644e3abe1e81539fe3d83, State: Running, Role: LEADER
I20260619 14:58:04.602070 20329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfa8c02078644e3abe1e81539fe3d83" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40867 } }
I20260619 14:58:04.602478 20324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:58:04.603489 20331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bfa8c02078644e3abe1e81539fe3d83. Latest consensus state: current_term: 1 leader_uuid: "8bfa8c02078644e3abe1e81539fe3d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfa8c02078644e3abe1e81539fe3d83" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40867 } } }
I20260619 14:58:04.603489 20330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bfa8c02078644e3abe1e81539fe3d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bfa8c02078644e3abe1e81539fe3d83" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40867 } } }
I20260619 14:58:04.603641 20331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:04.603714 20330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83 [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:04.604440 20339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:58:04.607031 20339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:58:04.611574 20339 catalog_manager.cc:1347] Generated new cluster ID: f81d6b21955f452c95a160b5db9eddf2
I20260619 14:58:04.611634 20339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:58:04.631778 20339 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:58:04.632036 20339 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:58:04.638661 20339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bfa8c02078644e3abe1e81539fe3d83: Generated new TSK 0
I20260619 14:58:04.639201 20339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:58:04.649893 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40867
--builtin_ntp_servers=127.10.71.148:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:58:04.751142 20348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:04.751340 20348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:04.751374 20348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:04.755141 20348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:04.755182 20348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:04.755228 20348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:58:04.759430 20348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:04.760183 20348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:04.761312 20348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:04.769807 20356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.769882 20353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.769809 20354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.770062 20348 server_base.cc:1034] running on GCE node
I20260619 14:58:04.770542 20348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:04.771107 20348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:04.772343 20348 hybrid_clock.cc:648] HybridClock initialized: now 1781881084772318 us; error 33 us; skew 500 ppm
I20260619 14:58:04.774125 20348 webserver.cc:460] Webserver started at http://127.10.71.129:42027/ using document root <none> and password file <none>
I20260619 14:58:04.774614 20348 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:04.774700 20348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:04.774911 20348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:04.776800 20348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "699684a01204474a9b6b49dde686da52"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "3c9c0ab6b9f6fb904e9b0887c2af2b53"
server_key_iv: "a024f106d4de5b19076d14f55ed60f7f"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.777344 20348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "699684a01204474a9b6b49dde686da52"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "3c9c0ab6b9f6fb904e9b0887c2af2b53"
server_key_iv: "a024f106d4de5b19076d14f55ed60f7f"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.780997 20348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:58:04.783069 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.784106 20348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:58:04.784235 20348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "699684a01204474a9b6b49dde686da52"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "3c9c0ab6b9f6fb904e9b0887c2af2b53"
server_key_iv: "a024f106d4de5b19076d14f55ed60f7f"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.784354 20348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:04.788187 20348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:04.788702 20348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:04.788825 20348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:04.789234 20348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:04.789906 20348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:04.789944 20348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.789976 20348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:04.789995 20348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.797483 20348 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:35409
I20260619 14:58:04.797508 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:35409 every 8 connection(s)
I20260619 14:58:04.798171 20348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:58:04.803117 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.799694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:57063 (local address 127.10.71.190:40867)
0619 14:58:04.799844 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.799849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.800722 (+ 873us) server_negotiation.cc:407] Connection header received
0619 14:58:04.801738 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.801740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.801772 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.801836 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.802737 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:04.802740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:04.802742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:04.802753 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:04.802774 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:04.802787 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:04.802864 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:04.802985 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:58:04.803016 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:58:04.803889 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.800018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40867 (local address 127.10.71.129:57063)
0619 14:58:04.800595 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:04.800632 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:04.801586 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:04.801964 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:04.801969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:04.802041 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:04.802488 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:58:04.802489 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:04.802521 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:58:04.802578 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:04.802871 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:04.802878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:04.802883 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:58:04.803001 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:58:04.803203 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260619 14:58:04.804955 20476 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40867
I20260619 14:58:04.805147 20476 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:04.805264 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20348
I20260619 14:58:04.805365 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:58:04.805509 20476 heartbeater.cc:511] Master 127.10.71.190:40867 requested a full tablet report, sending...
I20260619 14:58:04.805667 10526 external_mini_cluster.cc:1428] Setting key 16b6209c93dcd1ba64b122ade8850179
I20260619 14:58:04.807031 20287 ts_manager.cc:194] Registered new tserver with Master: 699684a01204474a9b6b49dde686da52 (127.10.71.129:35409)
I20260619 14:58:04.807617 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40867
--builtin_ntp_servers=127.10.71.148:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:58:04.909746 20479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:04.909925 20479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:04.909956 20479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:04.913410 20479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:04.913448 20479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:04.913494 20479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:58:04.917557 20479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:04.918242 20479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:04.919239 20479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:04.927410 20484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.927366 20487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:04.927367 20485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:04.927417 20479 server_base.cc:1034] running on GCE node
I20260619 14:58:04.927934 20479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:04.928503 20479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:04.929679 20479 hybrid_clock.cc:648] HybridClock initialized: now 1781881084929670 us; error 29 us; skew 500 ppm
I20260619 14:58:04.931241 20479 webserver.cc:460] Webserver started at http://127.10.71.130:39191/ using document root <none> and password file <none>
I20260619 14:58:04.931679 20479 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:04.931731 20479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:04.931865 20479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:04.933650 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "ce56e11f4ba34e98a9af8a59de555e93"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "62bb500e4d41a4f961d41fd0d080bafd"
server_key_iv: "c532460655bae07c45e3fb112c9fd4c1"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.934100 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "ce56e11f4ba34e98a9af8a59de555e93"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "62bb500e4d41a4f961d41fd0d080bafd"
server_key_iv: "c532460655bae07c45e3fb112c9fd4c1"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.937278 20479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:58:04.939378 20493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.940495 20479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:58:04.940634 20479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "ce56e11f4ba34e98a9af8a59de555e93"
format_stamp: "Formatted at 2026-06-19 14:58:04 on dist-test-slave-3qdh"
server_key: "62bb500e4d41a4f961d41fd0d080bafd"
server_key_iv: "c532460655bae07c45e3fb112c9fd4c1"
server_key_version: "encryptionkey@0"
I20260619 14:58:04.940747 20479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:04.944972 20479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:04.945508 20479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:04.945688 20479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:04.946200 20479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:04.947032 20479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:04.947110 20479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.947185 20479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:04.947233 20479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:04.955663 20479 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:35039
I20260619 14:58:04.955691 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:35039 every 8 connection(s)
I20260619 14:58:04.956513 20479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:58:04.961338 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.958034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:36683 (local address 127.10.71.190:40867)
0619 14:58:04.958150 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:04.958153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:04.958953 (+ 800us) server_negotiation.cc:407] Connection header received
0619 14:58:04.959970 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:04.959972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:04.960004 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:04.960074 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:04.960969 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:04.960971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:04.960973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:04.960982 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:04.961011 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:04.961017 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:04.961096 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:04.961198 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:58:04.961229 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:58:04.962035 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:04.958298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40867 (local address 127.10.71.130:36683)
0619 14:58:04.958837 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:04.958869 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:04.959809 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:04.960189 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:04.960195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:04.960263 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:04.960760 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:58:04.960761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:04.960792 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:58:04.960845 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:04.961096 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:04.961101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:04.961104 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:58:04.961207 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:58:04.961369 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
I20260619 14:58:04.962962 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20479
I20260619 14:58:04.962998 20607 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40867
I20260619 14:58:04.963119 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:58:04.963225 20607 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:04.963389 10526 external_mini_cluster.cc:1428] Setting key 48917a24676b8ed34bfe35fafaaa90d7
I20260619 14:58:04.963557 20607 heartbeater.cc:511] Master 127.10.71.190:40867 requested a full tablet report, sending...
I20260619 14:58:04.964499 20288 ts_manager.cc:194] Registered new tserver with Master: ce56e11f4ba34e98a9af8a59de555e93 (127.10.71.130:35039)
I20260619 14:58:04.965196 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40867
--builtin_ntp_servers=127.10.71.148:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:58:05.063016 20610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:05.063212 20610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:05.063244 20610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:05.067094 20610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:05.067138 20610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:05.067188 20610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:58:05.071620 20610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:05.072471 20610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:05.073633 20610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:05.082007 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:05.082011 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:05.082015 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:05.082235 20610 server_base.cc:1034] running on GCE node
I20260619 14:58:05.082682 20610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:05.083182 20610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:05.084363 20610 hybrid_clock.cc:648] HybridClock initialized: now 1781881085084336 us; error 36 us; skew 500 ppm
I20260619 14:58:05.085911 20610 webserver.cc:460] Webserver started at http://127.10.71.131:44769/ using document root <none> and password file <none>
I20260619 14:58:05.086398 20610 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:05.086454 20610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:05.086728 20610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:05.088722 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "d13edb24bc554b5c942bf570ba6436dd"
format_stamp: "Formatted at 2026-06-19 14:58:05 on dist-test-slave-3qdh"
server_key: "e98e4af943a69b4b25dbfd9c42cf3249"
server_key_iv: "a417a48d44fc90e9a338ff127c34cda5"
server_key_version: "encryptionkey@0"
I20260619 14:58:05.089213 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "d13edb24bc554b5c942bf570ba6436dd"
format_stamp: "Formatted at 2026-06-19 14:58:05 on dist-test-slave-3qdh"
server_key: "e98e4af943a69b4b25dbfd9c42cf3249"
server_key_iv: "a417a48d44fc90e9a338ff127c34cda5"
server_key_version: "encryptionkey@0"
I20260619 14:58:05.092193 20610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:58:05.094297 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:05.095422 20610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:58:05.095598 20610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "d13edb24bc554b5c942bf570ba6436dd"
format_stamp: "Formatted at 2026-06-19 14:58:05 on dist-test-slave-3qdh"
server_key: "e98e4af943a69b4b25dbfd9c42cf3249"
server_key_iv: "a417a48d44fc90e9a338ff127c34cda5"
server_key_version: "encryptionkey@0"
I20260619 14:58:05.095717 20610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:05.099920 20610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:05.100510 20610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:05.100689 20610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:05.101150 20610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:05.101969 20610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:05.102039 20610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:05.102109 20610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:05.102154 20610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:05.111032 20610 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:39435
I20260619 14:58:05.111050 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:39435 every 8 connection(s)
I20260619 14:58:05.111760 20610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:58:05.116947 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:05.113291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:52029 (local address 127.10.71.190:40867)
0619 14:58:05.113409 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:05.113413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:05.114228 (+ 815us) server_negotiation.cc:407] Connection header received
0619 14:58:05.115340 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:05.115343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:05.115385 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:05.115483 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:05.116496 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:05.116500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:05.116503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:58:05.116514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:05.116535 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:58:05.116551 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:05.116645 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:05.116783 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:58:05.116824 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:58:05.117627 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:05.113564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:40867 (local address 127.10.71.131:52029)
0619 14:58:05.114109 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:05.114140 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:05.115152 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:05.115622 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:05.115628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:05.115702 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:05.116221 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:58:05.116222 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:05.116255 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:58:05.116368 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:05.116633 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:05.116641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:05.116646 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:58:05.116809 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:58:05.117008 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260619 14:58:05.118588 20738 heartbeater.cc:347] Connected to a master server at 127.10.71.190:40867
I20260619 14:58:05.118774 20738 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:05.119091 20738 heartbeater.cc:511] Master 127.10.71.190:40867 requested a full tablet report, sending...
I20260619 14:58:05.120083 20288 ts_manager.cc:194] Registered new tserver with Master: d13edb24bc554b5c942bf570ba6436dd (127.10.71.131:39435)
I20260619 14:58:05.120556 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20610
I20260619 14:58:05.120673 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:58:05.120946 10526 external_mini_cluster.cc:1428] Setting key c3a460d3698cb1610ff1d7b668e51863
I20260619 14:58:05.122831 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:58:05.125489 20327 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:05.124958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.10.71.190:40867)
0619 14:58:05.125103 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:05.125108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:05.125123 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:58:05.125156 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:05.125160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:05.125191 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:05.125264 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:05.125413 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59934: BlockingRecv error: recv got EOF from 127.0.0.1:59934 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:58:05.125715 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20348
I20260619 14:58:05.131454 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20479
I20260619 14:58:05.136523 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20610
I20260619 14:58:05.141494 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20257
2026-06-19T14:58:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (732 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:58:05 dist-test-slave-3qdh krb5kdc[20748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:58:05 dist-test-slave-3qdh krb5kdc[20748](info): set up 2 sockets
Jun 19 14:58:05 dist-test-slave-3qdh krb5kdc[20748](info): commencing operation
krb5kdc: starting...
W20260619 14:58:06.100848 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:58:06.201112 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:36643
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:06.295907 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:06.296118 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:06.296157 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:06.299644 20761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:58:06.299690 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:06.299707 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:06.299722 20761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:58:06.299736 20761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:58:06.304129 20761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:36643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:06.305054 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:06.305756 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:58:06.313253 20761 server_base.cc:1034] running on GCE node
W20260619 14:58:06.313215 20766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:06.313273 20769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:06.313215 20767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:06.313815 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:06.314630 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:06.315801 20761 hybrid_clock.cc:648] HybridClock initialized: now 1781881086315777 us; error 33 us; skew 500 ppm
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:06.318332 20761 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:58:06.319028 20761 webserver.cc:460] Webserver started at http://127.10.71.190:37273/ using document root <none> and password file <none>
I20260619 14:58:06.319427 20761 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:06.319476 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:06.319621 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:06.321223 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "c429ec7b9e4517cbb8774761ce8c25df"
server_key_iv: "6c281dfb4a8e35aba2a1d6f735b5d570"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.321666 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "c429ec7b9e4517cbb8774761ce8c25df"
server_key_iv: "6c281dfb4a8e35aba2a1d6f735b5d570"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.324656 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:58:06.326746 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.327891 20761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:58:06.327980 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "c429ec7b9e4517cbb8774761ce8c25df"
server_key_iv: "6c281dfb4a8e35aba2a1d6f735b5d570"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.328058 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:06.342661 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:06.344892 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:06.345029 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:06.351390 20761 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:36643
I20260619 14:58:06.351610 20828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:36643 every 8 connection(s)
I20260619 14:58:06.352201 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:58:06.354557 20829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:06.355062 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20761
I20260619 14:58:06.355154 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:58:06.355419 10526 external_mini_cluster.cc:1428] Setting key ee03c651b46f3de1925d6d4be4a60ff5
I20260619 14:58:06.360467 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e: Bootstrap starting.
I20260619 14:58:06.362553 20829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:06.363245 20829 log.cc:824] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:06.364877 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e: No bootstrap required, opened a new log
I20260619 14:58:06.368634 20829 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36643 } }
I20260619 14:58:06.368847 20829 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:06.368922 20829 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bc81c7f82d4e10a52fc7a29060bb4e, State: Initialized, Role: FOLLOWER
I20260619 14:58:06.369339 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.356720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.10.71.190:36643)
0619 14:58:06.357204 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.357219 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.357253 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:58:06.357920 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.357939 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.358292 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.358591 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:06.359779 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.361019 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.361839 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.362434 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.362577 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.364884 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:06.364889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:06.364899 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:06.364921 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:06.366443 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.366953 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.366958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.366961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.367070 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.367356 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.367358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.367359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.367670 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:06.367830 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.368237 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.368560 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260619 14:58:06.369303 20829 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36643 } }
I20260619 14:58:06.369501 20829 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:58:06.369611 20829 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:58:06.369725 20829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:06.370504 20829 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36643 } }
I20260619 14:58:06.370795 20829 leader_election.cc:302] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4bc81c7f82d4e10a52fc7a29060bb4e; no voters:
I20260619 14:58:06.371062 20829 leader_election.cc:288] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:58:06.371155 20834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:58:06.371359 20834 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [term 1 LEADER]: Becoming Leader. State: Replica: f4bc81c7f82d4e10a52fc7a29060bb4e, State: Running, Role: LEADER
I20260619 14:58:06.371610 20834 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36643 } }
I20260619 14:58:06.372110 20829 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:58:06.373121 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4bc81c7f82d4e10a52fc7a29060bb4e. Latest consensus state: current_term: 1 leader_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36643 } } }
I20260619 14:58:06.373239 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:06.373493 20835 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4bc81c7f82d4e10a52fc7a29060bb4e" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 36643 } } }
I20260619 14:58:06.373587 20835 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:06.373672 20843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:58:06.376467 20843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:58:06.381577 20843 catalog_manager.cc:1347] Generated new cluster ID: c6f8fdbdf81a4aeaa950a9aac11bb202
I20260619 14:58:06.381649 20843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:58:06.397709 20843 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:58:06.398626 20843 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:58:06.403896 20843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e: Generated new TSK 0
I20260619 14:58:06.404448 20843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:58:06.437214 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36643
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:06.535060 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:06.535274 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:06.535338 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:06.538936 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:06.539011 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:06.539093 20855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:58:06.543479 20855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:06.544269 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:06.545094 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:06.553588 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:06.553602 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:06.553819 20855 server_base.cc:1034] running on GCE node
W20260619 14:58:06.553587 20860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:06.554226 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:06.554772 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:06.555960 20855 hybrid_clock.cc:648] HybridClock initialized: now 1781881086555933 us; error 52 us; skew 500 ppm
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:06.558490 20855 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:58:06.559262 20855 webserver.cc:460] Webserver started at http://127.10.71.129:46309/ using document root <none> and password file <none>
I20260619 14:58:06.559733 20855 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:06.559814 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:06.560017 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:06.561753 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "ccd9517833b4456dbde9d4ee9439b5d7"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "23984dd829d37b20b6dcb2a4c5d0c99b"
server_key_iv: "1048764f1948c86b607070a0e602aa97"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.562229 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "ccd9517833b4456dbde9d4ee9439b5d7"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "23984dd829d37b20b6dcb2a4c5d0c99b"
server_key_iv: "1048764f1948c86b607070a0e602aa97"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.565218 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:58:06.567317 20870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.568498 20855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:58:06.568617 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "ccd9517833b4456dbde9d4ee9439b5d7"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "23984dd829d37b20b6dcb2a4c5d0c99b"
server_key_iv: "1048764f1948c86b607070a0e602aa97"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.568715 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:06.577656 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:06.579888 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:06.580055 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:06.580606 20855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:06.581254 20855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:06.581318 20855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.581373 20855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:06.581413 20855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.589150 20855 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:37267
I20260619 14:58:06.589174 20983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:37267 every 8 connection(s)
I20260619 14:58:06.589835 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:58:06.591936 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20855
I20260619 14:58:06.592049 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:58:06.592429 10526 external_mini_cluster.cc:1428] Setting key 09b267f203f9510a9cf6988eeffae3b1
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:58:06.602562 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.591291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:40841 (local address 127.10.71.190:36643)
0619 14:58:06.591402 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.591405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.592112 (+ 707us) server_negotiation.cc:407] Connection header received
0619 14:58:06.593424 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.593428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.593468 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.593551 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:06.594867 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.595454 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.596584 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.596816 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.596929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.599096 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:06.599099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:06.599102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:06.599133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:06.600620 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.601125 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.601128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.601130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.601238 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.601545 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.601549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.601551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.601723 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:06.601822 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.602297 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.602412 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:58:06.603302 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.591510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36643 (local address 127.10.71.129:40841)
0619 14:58:06.591960 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:06.592010 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:06.593258 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:06.593724 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:06.593730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:06.594156 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:06.594729 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.594746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.595570 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.595572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.596453 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.596462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.596956 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.596960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.597129 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.597656 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:06.597669 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:06.598917 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:06.600736 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:06.600741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:06.600752 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:06.600998 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:06.601323 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:06.601327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:06.601329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:06.601454 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:06.601823 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:06.601829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:06.602084 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 14:58:06.602254 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:58:06.602514 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":96,"threads_started":1}
I20260619 14:58:06.604358 20984 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36643
I20260619 14:58:06.604566 20984 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:06.604955 20984 heartbeater.cc:511] Master 127.10.71.190:36643 requested a full tablet report, sending...
I20260619 14:58:06.606277 20793 ts_manager.cc:194] Registered new tserver with Master: ccd9517833b4456dbde9d4ee9439b5d7 (127.10.71.129:37267)
I20260619 14:58:06.607343 20793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:40841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:58:06.616212 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36643
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:06.710558 20989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:06.710726 20989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:06.710758 20989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:06.714239 20989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:06.714288 20989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:06.714370 20989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:58:06.719053 20989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:06.719772 20989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:06.720592 20989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:58:06.728188 20989 server_base.cc:1034] running on GCE node
W20260619 14:58:06.728160 20994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:06.728153 20995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:06.728160 20997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:06.728719 20989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:06.729192 20989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:06.730337 20989 hybrid_clock.cc:648] HybridClock initialized: now 1781881086730323 us; error 25 us; skew 500 ppm
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:06.732841 20989 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:58:06.733592 20989 webserver.cc:460] Webserver started at http://127.10.71.130:37863/ using document root <none> and password file <none>
I20260619 14:58:06.734086 20989 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:06.734140 20989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:06.734361 20989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:06.736119 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "b0fd5bfa03e5427b88f92eb060f75b1e"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "3cc9b2b827fb915a3431237e284e62fd"
server_key_iv: "e210dc2789b3ed291a2a740aeb8773bd"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.736678 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "b0fd5bfa03e5427b88f92eb060f75b1e"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "3cc9b2b827fb915a3431237e284e62fd"
server_key_iv: "e210dc2789b3ed291a2a740aeb8773bd"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.739744 20989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:58:06.741770 21004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.742806 20989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:58:06.742918 20989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "b0fd5bfa03e5427b88f92eb060f75b1e"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "3cc9b2b827fb915a3431237e284e62fd"
server_key_iv: "e210dc2789b3ed291a2a740aeb8773bd"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.743005 20989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:06.763257 20989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:06.765708 20989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:06.765892 20989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:06.766398 20989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:06.767094 20989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:06.767174 20989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.767242 20989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:06.767288 20989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.776280 20989 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:38141
I20260619 14:58:06.776348 21117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:38141 every 8 connection(s)
I20260619 14:58:06.777032 20989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:58:06.780812 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 20989
I20260619 14:58:06.780937 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:58:06.781250 10526 external_mini_cluster.cc:1428] Setting key 16e398920dd1bb701e1b0954026448d7
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:06.788916 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.778449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:48175 (local address 127.10.71.190:36643)
0619 14:58:06.778611 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.778616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.779218 (+ 602us) server_negotiation.cc:407] Connection header received
0619 14:58:06.780276 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.780278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.780336 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.780402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:06.781699 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.782237 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.783306 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.783527 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.783654 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.785727 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:06.785732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:06.785739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:06.785757 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:06.787134 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.787607 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.787611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.787613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.787683 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.787996 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.788000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.788002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.788152 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:06.788245 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.788672 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.788771 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:58:06.789552 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.778730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36643 (local address 127.10.71.130:48175)
0619 14:58:06.779090 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:06.779125 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:06.780116 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:06.780530 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:06.780535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:06.781015 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:06.781550 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.781571 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.782371 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.782374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.783188 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.783197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.783682 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.783686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.783832 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.784358 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:06.784370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:06.785587 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:06.787279 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:06.787284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:06.787295 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:06.787510 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:06.787765 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:06.787770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:06.787771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:06.787909 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:06.788244 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:06.788250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:06.788471 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:58:06.788629 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:58:06.788874 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260619 14:58:06.790483 21118 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36643
I20260619 14:58:06.790678 21118 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:06.791038 21118 heartbeater.cc:511] Master 127.10.71.190:36643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 14:58:06.791970 20793 ts_manager.cc:194] Registered new tserver with Master: b0fd5bfa03e5427b88f92eb060f75b1e (127.10.71.130:38141)
I20260619 14:58:06.792822 20793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:48175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:58:06.804553 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36643
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:06.899441 21123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:06.899617 21123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:06.899650 21123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:06.902897 21123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:06.902938 21123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:06.902982 21123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:58:06.907581 21123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:06.908366 21123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:06.909099 21123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:06.916728 21131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:06.916708 21123 server_base.cc:1034] running on GCE node
W20260619 14:58:06.916702 21128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:06.916704 21129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:06.917339 21123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:06.917799 21123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:06.918954 21123 hybrid_clock.cc:648] HybridClock initialized: now 1781881086918921 us; error 45 us; skew 500 ppm
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:06.921204 21123 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:58:06.921864 21123 webserver.cc:460] Webserver started at http://127.10.71.131:38289/ using document root <none> and password file <none>
I20260619 14:58:06.922293 21123 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:06.922369 21123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:06.922549 21123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:58:06.924039 21123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "8c14be5a238d4b9487eecd15b68fa4cf"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "79612049f0ed9c1ee69ae939e9d26e62"
server_key_iv: "0ec5973c79d77e54ce459b79da10b269"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.924552 21123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "8c14be5a238d4b9487eecd15b68fa4cf"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "79612049f0ed9c1ee69ae939e9d26e62"
server_key_iv: "0ec5973c79d77e54ce459b79da10b269"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.927609 21123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:58:06.929783 21138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.930694 21123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:58:06.930817 21123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "8c14be5a238d4b9487eecd15b68fa4cf"
format_stamp: "Formatted at 2026-06-19 14:58:06 on dist-test-slave-3qdh"
server_key: "79612049f0ed9c1ee69ae939e9d26e62"
server_key_iv: "0ec5973c79d77e54ce459b79da10b269"
server_key_version: "encryptionkey@0"
I20260619 14:58:06.930903 21123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:06.938621 21123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:06.940744 21123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:06.940912 21123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:06.941350 21123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:06.941944 21123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:06.942005 21123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.942060 21123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:06.942091 21123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:06.950067 21123 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:34617
I20260619 14:58:06.950089 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:34617 every 8 connection(s)
I20260619 14:58:06.950790 21123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:58:06 dist-test-slave-3qdh krb5kdc[20748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881086, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:06.958956 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 21123
I20260619 14:58:06.959071 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:58:06.959304 10526 external_mini_cluster.cc:1428] Setting key 534b0a63dac7b634ccb0c313c3f84448
I20260619 14:58:06.962339 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.952248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:59681 (local address 127.10.71.190:36643)
0619 14:58:06.952384 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.952387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.953220 (+ 833us) server_negotiation.cc:407] Connection header received
0619 14:58:06.953967 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.953971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.954024 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.954125 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:06.955127 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.955651 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.956780 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.956995 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.957101 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.959259 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:06.959261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:06.959263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:06.959280 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:06.960798 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.961320 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.961322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.961323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.961385 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.961631 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.961633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.961634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.961764 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:06.961825 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.962175 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.962245 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:58:06.962960 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.952567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:36643 (local address 127.10.71.131:59681)
0619 14:58:06.953100 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:06.953132 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:06.953797 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:06.954297 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:06.954303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:06.954607 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:06.954989 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.955000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.955765 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.955771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.956664 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.956673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.957145 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.957153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.957317 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.957812 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:06.957826 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:06.959112 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:06.960898 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:06.960903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:06.960914 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:06.961223 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:06.961451 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:06.961453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:06.961454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:06.961559 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:06.961836 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:06.961839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:06.961999 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 14:58:06.962133 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:58:06.962363 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":88,"threads_started":1}
I20260619 14:58:06.963819 21252 heartbeater.cc:347] Connected to a master server at 127.10.71.190:36643
I20260619 14:58:06.963994 21252 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:06.964401 21252 heartbeater.cc:511] Master 127.10.71.190:36643 requested a full tablet report, sending...
I20260619 14:58:06.965246 20793 ts_manager.cc:194] Registered new tserver with Master: 8c14be5a238d4b9487eecd15b68fa4cf (127.10.71.131:34617)
I20260619 14:58:06.965775 20793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:59681
I20260619 14:58:06.972853 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:58:06.981177 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.974948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.10.71.190:36643)
0619 14:58:06.975048 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.975052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.975335 (+ 283us) server_negotiation.cc:407] Connection header received
0619 14:58:06.975429 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.975431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.975461 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.975518 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:06.976101 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.976627 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.977403 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.977610 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.977676 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.978764 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:06.978767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:06.978768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:06.978786 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:06.979959 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.980280 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.980282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.980283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.980384 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:06.980586 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:06.980588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:06.980589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:06.980767 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:06.980854 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.981012 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.981086 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:58:06.983536 20793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:58:06.986243 20793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:58:06.999245 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.994265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:38141 (local address 127.0.0.1:42512)
0619 14:58:06.994584 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:06.994600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:06.994680 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:06.995313 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:06.995315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:06.995330 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:06.995448 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.995456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.996706 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.996710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.997865 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.997874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.998810 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.998927 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.998961 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:58:06.999011 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 14:58:06.999127 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":32,"thread_start_us":80,"threads_started":1}
I20260619 14:58:06.999384 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.993878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.10.71.131:34617)
0619 14:58:06.994227 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.994238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.994442 (+ 204us) server_negotiation.cc:407] Connection header received
0619 14:58:06.994697 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.994702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.994880 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.995006 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:06.995508 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.996415 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998179 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.998643 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998762 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.998885 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.999135 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.999223 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260619 14:58:06.999646 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.993666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37267 (local address 127.0.0.1:47414)
0619 14:58:06.994293 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:06.994337 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:06.995050 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:06.995682 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:06.995684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:06.995701 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:06.995792 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.995798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.997042 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.997047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.998084 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.998093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.998981 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.999072 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.999098 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:58:06.999177 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:58:06.999233 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":37,"thread_start_us":65,"threads_started":1}
I20260619 14:58:06.999681 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.993727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:34617 (local address 127.0.0.1:60930)
0619 14:58:06.994356 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:06.994370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:06.994513 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:06.995198 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:06.995204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:06.995225 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:06.995379 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.995385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.996544 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.996547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.998071 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:06.998079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.998760 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:06.998868 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.998943 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:58:06.999049 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:58:06.999577 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":586,"thread_start_us":82,"threads_started":1}
I20260619 14:58:06.999696 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.994519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.10.71.130:38141)
0619 14:58:06.994895 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.994902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.994919 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:58:06.995002 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.995007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.995176 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.995331 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:06.995553 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.996561 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.997994 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.998691 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.999308 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.999434 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.999476 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.999567 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":97,"threads_started":1}
I20260619 14:58:07.000139 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:06.993754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.10.71.129:37267)
0619 14:58:06.994308 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:06.994320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:06.994985 (+ 665us) server_negotiation.cc:407] Connection header received
0619 14:58:06.995209 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:06.995215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:06.995383 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:06.995509 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:06.995911 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.996919 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998203 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:06.998891 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:06.998978 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:06.999113 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:06.999200 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:58:06.999288 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":77,"threads_started":1}
I20260619 14:58:07.002101 20918 tablet_service.cc:1467] Processing CreateTablet for tablet f6c2908b9b6145c0b9849996244075ea (DEFAULT_TABLE table=test-table [id=34fc3e8d307d47d88f243bcfc32bd7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:58:07.002238 21186 tablet_service.cc:1467] Processing CreateTablet for tablet f6c2908b9b6145c0b9849996244075ea (DEFAULT_TABLE table=test-table [id=34fc3e8d307d47d88f243bcfc32bd7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:58:07.002238 21052 tablet_service.cc:1467] Processing CreateTablet for tablet f6c2908b9b6145c0b9849996244075ea (DEFAULT_TABLE table=test-table [id=34fc3e8d307d47d88f243bcfc32bd7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:58:07.002962 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c2908b9b6145c0b9849996244075ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:07.003103 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c2908b9b6145c0b9849996244075ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:07.003109 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c2908b9b6145c0b9849996244075ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:07.008684 21269 tablet_bootstrap.cc:492] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: Bootstrap starting.
I20260619 14:58:07.009325 21270 tablet_bootstrap.cc:492] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Bootstrap starting.
I20260619 14:58:07.011128 21269 tablet_bootstrap.cc:654] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:07.011128 21270 tablet_bootstrap.cc:654] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:07.011950 21269 log.cc:824] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:07.012105 21270 log.cc:824] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:07.012578 21271 tablet_bootstrap.cc:492] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Bootstrap starting.
I20260619 14:58:07.013622 21269 tablet_bootstrap.cc:492] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: No bootstrap required, opened a new log
I20260619 14:58:07.013698 21270 tablet_bootstrap.cc:492] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: No bootstrap required, opened a new log
I20260619 14:58:07.013815 21269 ts_tablet_manager.cc:1397] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:58:07.013846 21270 ts_tablet_manager.cc:1397] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:58:07.014631 21271 tablet_bootstrap.cc:654] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:07.015285 21271 log.cc:824] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:07.016738 21269 raft_consensus.cc:348] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.016961 21269 raft_consensus.cc:374] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:07.017019 21269 raft_consensus.cc:729] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd9517833b4456dbde9d4ee9439b5d7, State: Initialized, Role: FOLLOWER
I20260619 14:58:07.017081 21271 tablet_bootstrap.cc:492] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: No bootstrap required, opened a new log
I20260619 14:58:07.017261 21271 ts_tablet_manager.cc:1397] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:58:07.017236 21270 raft_consensus.cc:348] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.017459 21270 raft_consensus.cc:374] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:07.017508 21270 raft_consensus.cc:729] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c14be5a238d4b9487eecd15b68fa4cf, State: Initialized, Role: FOLLOWER
I20260619 14:58:07.017477 21269 consensus_queue.cc:260] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.017876 21270 consensus_queue.cc:260] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.018369 21269 ts_tablet_manager.cc:1428] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260619 14:58:07.018632 21270 ts_tablet_manager.cc:1428] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:58:07.018688 20984 heartbeater.cc:503] Master 127.10.71.190:36643 was elected leader, sending a full tablet report...
I20260619 14:58:07.018909 21252 heartbeater.cc:503] Master 127.10.71.190:36643 was elected leader, sending a full tablet report...
I20260619 14:58:07.020103 21271 raft_consensus.cc:348] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.020319 21271 raft_consensus.cc:374] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:07.020381 21271 raft_consensus.cc:729] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0fd5bfa03e5427b88f92eb060f75b1e, State: Initialized, Role: FOLLOWER
I20260619 14:58:07.020754 21271 consensus_queue.cc:260] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.021502 21118 heartbeater.cc:503] Master 127.10.71.190:36643 was elected leader, sending a full tablet report...
I20260619 14:58:07.021600 21271 ts_tablet_manager.cc:1428] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260619 14:58:07.027768 21119 tablet.cc:2369] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:58:07.090793 20985 tablet.cc:2369] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:58:07.201638 21253 tablet.cc:2369] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:58:07.333667 21277 raft_consensus.cc:482] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:58:07.333885 21277 raft_consensus.cc:504] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.335050 21277 leader_election.cc:288] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccd9517833b4456dbde9d4ee9439b5d7 (127.10.71.129:37267), 8c14be5a238d4b9487eecd15b68fa4cf (127.10.71.131:34617)
I20260619 14:58:07.338757 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:07.335343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:49461 (local address 127.10.71.129:37267)
0619 14:58:07.335467 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:07.335472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:07.335591 (+ 119us) server_negotiation.cc:407] Connection header received
0619 14:58:07.335753 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:07.335756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:07.335810 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:07.335896 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:07.336190 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.336773 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.337852 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.338213 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.338339 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:07.338397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:07.338563 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 14:58:07.338627 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:58:07.338765 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:07.335385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:58151 (local address 127.10.71.131:34617)
0619 14:58:07.335517 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:07.335522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:07.335854 (+ 332us) server_negotiation.cc:407] Connection header received
0619 14:58:07.335986 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:07.335990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:07.336052 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:07.336139 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:07.336413 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.336899 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.337876 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.338247 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.338363 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:07.338412 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:07.338589 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:58:07.338648 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:58:07.338796 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:07.335242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:34617 (local address 127.10.71.130:58151)
0619 14:58:07.335771 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:07.335782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:07.335889 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:07.336149 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:07.336152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:07.336166 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:07.336281 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:07.336286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.337011 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.337016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:07.337737 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:07.337761 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.338358 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.338361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:07.338476 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:07.338491 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 14:58:07.338572 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:58:07.338635 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":241,"thread_start_us":127,"threads_started":1}
I20260619 14:58:07.338807 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:07.335242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37267 (local address 127.10.71.130:49461)
0619 14:58:07.335497 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:07.335510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:07.335645 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:07.335906 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:07.335909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:07.335926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:07.336074 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:07.336080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.336899 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.336903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:07.337711 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:07.337719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.338331 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.338335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:07.338429 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:07.338460 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:58:07.338548 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:58:07.338615 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260619 14:58:07.339454 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c2908b9b6145c0b9849996244075ea" candidate_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" is_pre_election: true
I20260619 14:58:07.339475 21206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c2908b9b6145c0b9849996244075ea" candidate_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" is_pre_election: true
I20260619 14:58:07.339701 20938 raft_consensus.cc:2454] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0fd5bfa03e5427b88f92eb060f75b1e in term 0.
I20260619 14:58:07.339788 21206 raft_consensus.cc:2454] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0fd5bfa03e5427b88f92eb060f75b1e in term 0.
I20260619 14:58:07.340126 21008 leader_election.cc:302] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0fd5bfa03e5427b88f92eb060f75b1e, ccd9517833b4456dbde9d4ee9439b5d7; no voters:
I20260619 14:58:07.340385 21277 raft_consensus.cc:2781] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:58:07.340483 21277 raft_consensus.cc:482] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:58:07.340543 21277 raft_consensus.cc:3037] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:07.341650 21277 raft_consensus.cc:504] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.341985 21277 leader_election.cc:288] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [CANDIDATE]: Term 1 election: Requested vote from peers ccd9517833b4456dbde9d4ee9439b5d7 (127.10.71.129:37267), 8c14be5a238d4b9487eecd15b68fa4cf (127.10.71.131:34617)
I20260619 14:58:07.342362 21206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c2908b9b6145c0b9849996244075ea" candidate_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c14be5a238d4b9487eecd15b68fa4cf"
I20260619 14:58:07.342378 20938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c2908b9b6145c0b9849996244075ea" candidate_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd9517833b4456dbde9d4ee9439b5d7"
I20260619 14:58:07.342480 21206 raft_consensus.cc:3037] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:07.342494 20938 raft_consensus.cc:3037] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:07.343429 20938 raft_consensus.cc:2454] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0fd5bfa03e5427b88f92eb060f75b1e in term 1.
I20260619 14:58:07.343492 21206 raft_consensus.cc:2454] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0fd5bfa03e5427b88f92eb060f75b1e in term 1.
I20260619 14:58:07.343670 21008 leader_election.cc:302] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0fd5bfa03e5427b88f92eb060f75b1e, ccd9517833b4456dbde9d4ee9439b5d7; no voters:
I20260619 14:58:07.343819 21277 raft_consensus.cc:2781] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:58:07.344048 21277 raft_consensus.cc:686] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 1 LEADER]: Becoming Leader. State: Replica: b0fd5bfa03e5427b88f92eb060f75b1e, State: Running, Role: LEADER
I20260619 14:58:07.344426 21277 consensus_queue.cc:237] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } }
I20260619 14:58:07.347502 20792 catalog_manager.cc:5495] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b0fd5bfa03e5427b88f92eb060f75b1e (127.10.71.130). New cstate: current_term: 1 leader_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0fd5bfa03e5427b88f92eb060f75b1e" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 38141 } health_report { overall_health: HEALTHY } } }
I20260619 14:58:07.358068 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:07.354775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.10.71.130:38141)
0619 14:58:07.354857 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:07.354861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:07.354925 (+ 64us) server_negotiation.cc:407] Connection header received
0619 14:58:07.355052 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:07.355055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:07.355094 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:07.355160 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:58:07.355373 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.355754 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.356919 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:07.357120 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:07.357196 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:07.357468 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:58:07.357767 (+ 299us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:58:07.357838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:07.357917 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 14:58:07.357956 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 14:58:07.363391 20938 raft_consensus.cc:1261] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 1 FOLLOWER]: Refusing update from remote peer b0fd5bfa03e5427b88f92eb060f75b1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:58:07.363974 21277 consensus_queue.cc:1035] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccd9517833b4456dbde9d4ee9439b5d7" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:58:07.364100 21206 raft_consensus.cc:1261] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 1 FOLLOWER]: Refusing update from remote peer b0fd5bfa03e5427b88f92eb060f75b1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:58:07.364641 21277 consensus_queue.cc:1035] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c14be5a238d4b9487eecd15b68fa4cf" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:58:07.378122 20792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:58:07.378264 20792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:58:07.380092 20792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4bc81c7f82d4e10a52fc7a29060bb4e: Sending DeleteTablet for 3 replicas of tablet f6c2908b9b6145c0b9849996244075ea
I20260619 14:58:07.380807 20918 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c2908b9b6145c0b9849996244075ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:58:07 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:47414
I20260619 14:58:07.380910 21052 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c2908b9b6145c0b9849996244075ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:58:07 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:42512
I20260619 14:58:07.381188 21293 tablet_replica.cc:330] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7: stopping tablet replica
I20260619 14:58:07.381198 21186 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c2908b9b6145c0b9849996244075ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:58:07 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:60930
I20260619 14:58:07.381445 21293 raft_consensus.cc:2229] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:58:07.381598 21295 tablet_replica.cc:330] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: stopping tablet replica
I20260619 14:58:07.381700 21293 raft_consensus.cc:2258] T f6c2908b9b6145c0b9849996244075ea P ccd9517833b4456dbde9d4ee9439b5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:58:07.381999 21295 raft_consensus.cc:2229] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:58:07.382164 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20855
I20260619 14:58:07.382774 21295 raft_consensus.cc:2258] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:58:07.384013 21295 ts_tablet_manager.cc:1905] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:58:07.384248 21294 tablet_replica.cc:330] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: stopping tablet replica
I20260619 14:58:07.384486 21294 raft_consensus.cc:2229] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:58:07.384745 21294 raft_consensus.cc:2258] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:58:07.385816 21294 ts_tablet_manager.cc:1905] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:58:07.387280 21295 ts_tablet_manager.cc:1918] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:58:07.387409 21295 log.cc:1196] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/f6c2908b9b6145c0b9849996244075ea
I20260619 14:58:07.387791 21295 ts_tablet_manager.cc:1939] T f6c2908b9b6145c0b9849996244075ea P 8c14be5a238d4b9487eecd15b68fa4cf: Deleting consensus metadata
I20260619 14:58:07.388264 21294 ts_tablet_manager.cc:1918] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:58:07.388381 21294 log.cc:1196] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/f6c2908b9b6145c0b9849996244075ea
I20260619 14:58:07.388700 21294 ts_tablet_manager.cc:1939] T f6c2908b9b6145c0b9849996244075ea P b0fd5bfa03e5427b88f92eb060f75b1e: Deleting consensus metadata
I20260619 14:58:07.389204 20777 catalog_manager.cc:4841] TS 8c14be5a238d4b9487eecd15b68fa4cf (127.10.71.131:34617): tablet f6c2908b9b6145c0b9849996244075ea (table test-table [id=34fc3e8d307d47d88f243bcfc32bd7a8]) successfully deleted
I20260619 14:58:07.389597 20780 catalog_manager.cc:4841] TS b0fd5bfa03e5427b88f92eb060f75b1e (127.10.71.130:38141): tablet f6c2908b9b6145c0b9849996244075ea (table test-table [id=34fc3e8d307d47d88f243bcfc32bd7a8]) successfully deleted
W20260619 14:58:07.389832 20780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:37267 (error 108)
I20260619 14:58:07.390067 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20989
I20260619 14:58:07.391198 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:07.390961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37267 (local address 127.0.0.1:47424)
0619 14:58:07.391075 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:07.391145 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:58:07.391376 20780 catalog_manager.cc:4584] TS ccd9517833b4456dbde9d4ee9439b5d7 (127.10.71.129:37267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6c2908b9b6145c0b9849996244075ea: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37267: connect: Connection refused (error 111)
I20260619 14:58:07.396422 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 21123
I20260619 14:58:07.400523 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 20761
2026-06-19T14:58:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2259 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:58:07 dist-test-slave-3qdh krb5kdc[21299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:58:07 dist-test-slave-3qdh krb5kdc[21299](info): set up 2 sockets
Jun 19 14:58:07 dist-test-slave-3qdh krb5kdc[21299](info): commencing operation
krb5kdc: starting...
W20260619 14:58:08.361197 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:58:08 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:58:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/rangeradmin_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/rangeradmin_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/rangerlookup_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/rangerlookup_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
I20260619 14:58:08.505146 10526 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskn16cqy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/postgres -l logfile start
2026-06-19 14:58:10.587 UTC [21329] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:58:10.587 UTC [21329] LOG: listening on IPv4 address "127.10.71.148", port 42385
2026-06-19 14:58:10.590 UTC [21329] LOG: listening on Unix socket "/tmp/.s.PGSQL.42385"
2026-06-19 14:58:10.621 UTC [21332] LOG: database system was shut down at 2026-06-19 14:58:09 UTC
2026-06-19 14:58:10.624 UTC [21329] LOG: database system is ready to accept connections
I20260619 14:58:11.504968 10526 mini_postgres.cc:96] Postgres bound to 42385
2026-06-19 14:58:11.510 UTC [21340] FATAL: database "slave" does not exist
127.10.71.148:42385 - accepting connections
I20260619 14:58:11.511445 10526 mini_ranger.cc:161] Starting Ranger...
I20260619 14:58:11.525491 10526 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:58:11.899358 10526 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:58:11.899458 10526 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-admin
2026-06-19 14:58:12,154 [I] DB FLAVOR :POSTGRES
2026-06-19 14:58:12,155 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:58:12,155 [I] Checking connection..
2026-06-19 14:58:12,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:58:12,360 [I] Checking connection passed.
2026-06-19 14:58:12,360 [I] --------- Verifying version history table ---------
2026-06-19 14:58:12,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:58:12,606 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:58:12,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:58:12,845 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:58:12,846 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:58:12,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:58:13.084 UTC [21404] WARNING: there is no transaction in progress
2026-06-19 14:58:13,089 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:58:13,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:58:13,326 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:58:13,327 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:58:13,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:58:13,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:58:13,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3qdh.c.gcp-upstream.internal','N') ;"
2026-06-19 14:58:14,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:58:14,258 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:58:14,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:58:14,482 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:58:14,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:58:14,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:58:14,929 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:58:14,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:58:15.759 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.773 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.788 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.801 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.814 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.829 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.883 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.889 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.902 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.911 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.920 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.931 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.939 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.944 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.950 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:15.955 UTC [21512] WARNING: there is no transaction in progress
2026-06-19 14:58:16,494 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:58:16,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:58:16,723 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:58:16,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:58:16,937 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:58:16,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:58:17,160 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:58:17,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:58:17,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:58:17,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3qdh.c.gcp-upstream.internal';"
2026-06-19 14:58:17,790 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:58:17,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:58:17,999 [I] DB_PATCHES have already been applied
I20260619 14:58:18.005085 10526 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-admin:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskn16cqy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:58:18.005149 10526 mini_ranger.cc:193] Using host: 127.10.71.148
I20260619 14:58:18.008474 10526 mini_ranger.cc:227] Ranger admin URL: http://127.10.71.148:50303
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:58:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:58:19.054484 10526 mini_ranger.cc:160] Time spent starting Ranger: real 7.543s user 0.002s sys 0.007s
Jun 19, 2026 2:58:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:58:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:58:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:58:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:58:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:58:39.735654 10526 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/rangerkms_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/rangerkms_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.71.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.71.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:58:39 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:58:39.804998 10526 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:58:39.812814 10526 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:58:40.529031 10526 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:58:40.529130 10526 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms
I20260619 14:58:40.529232 10526 mini_ranger_kms.cc:227] Using postgres at 127.10.71.148:42385
2026-06-19 14:58:40,690 [I] DB FLAVOR :POSTGRES
2026-06-19 14:58:40,690 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:58:40,690 [I] Checking connection
2026-06-19 14:58:40,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:58:40,906 [I] connection success
2026-06-19 14:58:40,906 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:58:40,906 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:58:40,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:58:41,151 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:58:41,151 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:58:41,151 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:58:41,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:42385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:58:41,419 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:58:41.832027 10526 mini_ranger_kms.cc:310] Created kms service
I20260619 14:58:41.997560 10526 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:58:42.041596 10526 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:58:42 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:58:42.418824 10526 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:58:42.418977 10526 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms:/tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:58:42.419001 10526 mini_ranger_kms.cc:240] Using host: 127.10.71.148
I20260619 14:58:42.422943 10526 mini_ranger_kms.cc:276] Ranger KMS PID: 21692
I20260619 14:58:42.423087 10526 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.71.148:48483
Jun 19, 2026 2:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:58:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 14:58:44 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.71.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:44.494915 10526 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.690s user 0.012s sys 0.134s
I20260619 14:58:44.495009 10526 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:58:44.495123 10526 mini_ranger_kms.cc:401] 127.10.71.148:48483/kms/v1/keys
Jun 19 14:58:44 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.71.148@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
Jun 19 14:58:45 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.71.148@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
Jun 19 14:58:47 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:58:47.217118 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34151
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:34151
--ranger_config_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:48483/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:47.333444 21743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:47.333701 21743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:47.333735 21743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:47.337313 21743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:58:47.337358 21743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:47.337375 21743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:47.337389 21743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:58:47.337404 21743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:58:47.342499 21743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.71.148:48483/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:34151
--ranger_config_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:47.343421 21743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:47.344399 21743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:47.352560 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:47.352674 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:47.352560 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:47.352618 21743 server_base.cc:1034] running on GCE node
I20260619 14:58:47.353423 21743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:47.354403 21743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:47.355584 21743 hybrid_clock.cc:648] HybridClock initialized: now 1781881127355550 us; error 55 us; skew 500 ppm
Jun 19 14:58:47 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881127, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:47.362989 21743 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:58:47.363898 21743 webserver.cc:460] Webserver started at http://127.10.71.190:37615/ using document root <none> and password file <none>
I20260619 14:58:47.364447 21743 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:47.364503 21743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:47.364699 21743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:58:47 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881127, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:58:47.423018 21743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "594ea1da15124aa9ad12cea064fe7475"
format_stamp: "Formatted at 2026-06-19 14:58:47 on dist-test-slave-3qdh"
server_key: "102157c16a3d73e79ee697ea54790930"
server_key_iv: "84a80370655aba2878ac900cdd49a388"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:47.423609 21743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "594ea1da15124aa9ad12cea064fe7475"
format_stamp: "Formatted at 2026-06-19 14:58:47 on dist-test-slave-3qdh"
server_key: "102157c16a3d73e79ee697ea54790930"
server_key_iv: "84a80370655aba2878ac900cdd49a388"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:47.427232 21743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:58:47.443277 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:47.444643 21743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:58:47.444774 21743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "594ea1da15124aa9ad12cea064fe7475"
format_stamp: "Formatted at 2026-06-19 14:58:47 on dist-test-slave-3qdh"
server_key: "102157c16a3d73e79ee697ea54790930"
server_key_iv: "84a80370655aba2878ac900cdd49a388"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:47.444878 21743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:47.453931 21743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:47.456393 21743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:47.456553 21743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:47.463639 21743 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:34151
I20260619 14:58:47.463738 21810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:34151 every 8 connection(s)
I20260619 14:58:47.464560 21743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:58:47.467669 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:47.473734 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 21743
I20260619 14:58:47.473853 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:58:47.474126 10526 external_mini_cluster.cc:1428] Setting key 3a0b7deb401759cdb4ccbdc07e53231a
I20260619 14:58:47.474611 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475: Bootstrap starting.
I20260619 14:58:47.477236 21811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:47.478133 21811 log.cc:824] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:47.479769 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475: No bootstrap required, opened a new log
Jun 19 14:58:47 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:47.483647 21811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ea1da15124aa9ad12cea064fe7475" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34151 } }
I20260619 14:58:47.483901 21811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:47.483989 21811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594ea1da15124aa9ad12cea064fe7475, State: Initialized, Role: FOLLOWER
I20260619 14:58:47.484477 21811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ea1da15124aa9ad12cea064fe7475" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34151 } }
I20260619 14:58:47.484624 21811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:58:47.484704 21811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:58:47.484802 21811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:47.485572 21811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ea1da15124aa9ad12cea064fe7475" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34151 } }
I20260619 14:58:47.485879 21811 leader_election.cc:302] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594ea1da15124aa9ad12cea064fe7475; no voters:
I20260619 14:58:47.486195 21811 leader_election.cc:288] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:58:47.486301 21816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:58:47.486569 21816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [term 1 LEADER]: Becoming Leader. State: Replica: 594ea1da15124aa9ad12cea064fe7475, State: Running, Role: LEADER
I20260619 14:58:47.486909 21816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ea1da15124aa9ad12cea064fe7475" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34151 } }
I20260619 14:58:47.487389 21811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:58:47.488101 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594ea1da15124aa9ad12cea064fe7475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ea1da15124aa9ad12cea064fe7475" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34151 } } }
I20260619 14:58:47.489012 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:47.475342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.10.71.190:34151)
0619 14:58:47.475723 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:47.475736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:47.475765 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:58:47.476349 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:47.476373 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:47.476647 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:47.476937 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:47.477651 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:47.478622 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:47.479749 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:47.480436 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:47.480586 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:47.483538 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:47.483544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:47.483560 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:47.483590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:47.485842 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:47.486370 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:47.486375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:47.486380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:47.486502 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:47.486776 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:47.486779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:47.486780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:47.487204 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:47.487358 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:47.487790 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 14:58:47.488035 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260619 14:58:47.488277 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:47.489563 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594ea1da15124aa9ad12cea064fe7475. Latest consensus state: current_term: 1 leader_uuid: "594ea1da15124aa9ad12cea064fe7475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ea1da15124aa9ad12cea064fe7475" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34151 } } }
I20260619 14:58:47.489691 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475 [sys.catalog]: This master's current role is: LEADER
I20260619 14:58:47.489616 21811 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:58:47.495534 21830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:58:47.498333 21830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:58:47.504555 21830 catalog_manager.cc:1347] Generated new cluster ID: 064692b05cdf46ce85a9c303a49d9891
I20260619 14:58:47.504611 21830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:58:47.511018 21830 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:58:47.512149 21830 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:58:47.523000 21830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475: Generated new TSK 0
I20260619 14:58:47.523677 21830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:58:48.196101 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:48483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34151
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:48.343369 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:48.343740 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:48.343818 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:48.349861 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:48.349939 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:48.350040 21856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:58:48.357227 21856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.71.148:48483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:48.358433 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:48.359750 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:48.371039 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:48.371039 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:48.371505 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:48.372365 21856 server_base.cc:1034] running on GCE node
I20260619 14:58:48.372804 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:48.373453 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:48.374693 21856 hybrid_clock.cc:648] HybridClock initialized: now 1781881128374677 us; error 30 us; skew 500 ppm
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:48.377912 21856 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:58:48.378902 21856 webserver.cc:460] Webserver started at http://127.10.71.129:37553/ using document root <none> and password file <none>
I20260619 14:58:48.379432 21856 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:48.379519 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:48.379746 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:58:48.397765 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "e2edec1621c14bc081a1bbab18ca9db0"
format_stamp: "Formatted at 2026-06-19 14:58:48 on dist-test-slave-3qdh"
server_key: "51f550932ddcd0ad4be0c218281f6b48"
server_key_iv: "dfc558c97f2f1363419cdaba6bbc3ba4"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:48.398454 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "e2edec1621c14bc081a1bbab18ca9db0"
format_stamp: "Formatted at 2026-06-19 14:58:48 on dist-test-slave-3qdh"
server_key: "51f550932ddcd0ad4be0c218281f6b48"
server_key_iv: "dfc558c97f2f1363419cdaba6bbc3ba4"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:48.403388 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:58:48.417173 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:48.418578 21856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:58:48.418736 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "e2edec1621c14bc081a1bbab18ca9db0"
format_stamp: "Formatted at 2026-06-19 14:58:48 on dist-test-slave-3qdh"
server_key: "51f550932ddcd0ad4be0c218281f6b48"
server_key_iv: "dfc558c97f2f1363419cdaba6bbc3ba4"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:48.418854 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:48.442445 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:48.445755 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:48.445968 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:48.446614 21856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:48.447634 21856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:48.447724 21856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:48.448016 21856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:48.448099 21856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:48.457556 21856 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:40151
I20260619 14:58:48.458660 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:58:48.466065 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 21856
I20260619 14:58:48.466176 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:58:48.466440 10526 external_mini_cluster.cc:1428] Setting key 7bdf7ab907f6fa8761cae83202354162
I20260619 14:58:48.476449 21986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:40151 every 8 connection(s)
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:48.487919 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:48.461160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:35307 (local address 127.10.71.190:34151)
0619 14:58:48.461425 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:48.461432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:48.476335 (+ 14903us) server_negotiation.cc:407] Connection header received
0619 14:58:48.476856 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:48.476859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:48.476912 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:48.476989 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:48.478429 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.479167 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.481159 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.481427 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.481528 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:48.484323 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:48.484327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:48.484330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:48.484355 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:48.485944 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:48.486517 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:48.486520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:48.486522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:48.486603 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:48.486920 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:48.486922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:48.486924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:48.487089 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:48.487198 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:48.487626 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 14:58:48.487737 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:58:48.488903 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:48.461365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34151 (local address 127.10.71.129:35307)
0619 14:58:48.462372 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:48.462440 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:48.476474 (+ 14034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:48.477298 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:48.477310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:48.477673 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:48.478280 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:48.478298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.479652 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.479655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:48.481019 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:48.481028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.481688 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.481692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:48.481854 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:48.482488 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:48.482507 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:48.484154 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:48.486067 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:48.486075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:48.486089 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:48.486408 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:48.486693 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:48.486698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:48.486701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:48.486822 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:48.487194 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:48.487199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:48.487401 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 14:58:48.487564 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:58:48.487905 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":111,"threads_started":1}
I20260619 14:58:48.490281 21987 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34151
I20260619 14:58:48.490512 21987 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:48.491012 21987 heartbeater.cc:511] Master 127.10.71.190:34151 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
I20260619 14:58:48.492609 21775 ts_manager.cc:194] Registered new tserver with Master: e2edec1621c14bc081a1bbab18ca9db0 (127.10.71.129:40151)
I20260619 14:58:48.494055 21775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:35307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:58:48.513055 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:48483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34151
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:58:48.652439 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:48.652721 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:48.652796 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:48.658339 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:48.658425 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:48.658507 21995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:58:48.665433 21995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.71.148:48483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:48.666917 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:48.668102 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:58:48.679335 22004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:48.679608 21995 server_base.cc:1034] running on GCE node
W20260619 14:58:48.679894 22002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:48.680107 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:48.682413 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:48.682977 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:48.684185 21995 hybrid_clock.cc:648] HybridClock initialized: now 1781881128684159 us; error 41 us; skew 500 ppm
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:48.686798 21995 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:58:48.687731 21995 webserver.cc:460] Webserver started at http://127.10.71.130:36599/ using document root <none> and password file <none>
I20260619 14:58:48.688324 21995 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:48.688405 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:48.688613 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:58:48.702134 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "f30e4cea167842df9a264a615a690765"
format_stamp: "Formatted at 2026-06-19 14:58:48 on dist-test-slave-3qdh"
server_key: "326b98dc1acc303bed23b43e3fb4ba88"
server_key_iv: "c520e90698c396e7f8822b589c09a48d"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:48.703641 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "f30e4cea167842df9a264a615a690765"
format_stamp: "Formatted at 2026-06-19 14:58:48 on dist-test-slave-3qdh"
server_key: "326b98dc1acc303bed23b43e3fb4ba88"
server_key_iv: "c520e90698c396e7f8822b589c09a48d"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:48.713605 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.003s sys 0.000s
I20260619 14:58:48.741824 22011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:48.747498 21995 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.000s
I20260619 14:58:48.747701 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "f30e4cea167842df9a264a615a690765"
format_stamp: "Formatted at 2026-06-19 14:58:48 on dist-test-slave-3qdh"
server_key: "326b98dc1acc303bed23b43e3fb4ba88"
server_key_iv: "c520e90698c396e7f8822b589c09a48d"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:48.747918 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:48.768561 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:48.771864 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:48.772112 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:48.772826 21995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:48.773855 21995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:48.773953 21995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:48.774060 21995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:48.774125 21995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:48.784713 21995 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:40757
I20260619 14:58:48.785763 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:58:48.786365 22124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:40757 every 8 connection(s)
I20260619 14:58:48.791293 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 21995
I20260619 14:58:48.791409 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:58:48.791682 10526 external_mini_cluster.cc:1428] Setting key 1841b2f630e61a11c7099e14159e90a2
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:48.802141 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:48.787720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:50761 (local address 127.10.71.190:34151)
0619 14:58:48.787919 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:48.787923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:48.789037 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 14:58:48.790272 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:48.790276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:48.790333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:48.790424 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:48.791766 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.792591 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.794196 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.794496 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.794569 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:48.797522 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:48.797526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:48.797529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:48.797554 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:48.799327 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:48.800040 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:48.800046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:48.800048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:48.800147 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:48.800653 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:48.800658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:48.800660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:48.800919 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:48.801004 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:48.801777 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 14:58:48.801930 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:58:48.803925 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:48.788222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34151 (local address 127.10.71.130:50761)
0619 14:58:48.788834 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:48.788941 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:48.790030 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:48.790624 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:48.790633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:48.790970 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:48.791589 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:48.791611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.792757 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.792761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:48.794066 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:48.794077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:48.794762 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:48.794765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:48.794937 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:48.795564 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:48.795585 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:48.797350 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:48.799477 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:48.799484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:48.799501 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:48.799904 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:48.800277 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:48.800282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:48.800285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:48.800510 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:48.801098 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:48.801106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:48.801448 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 14:58:48.802322 (+ 874us) client_negotiation.cc:240] Negotiation successful
0619 14:58:48.802720 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":143,"threads_started":1}
I20260619 14:58:48.805205 22125 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 14:58:48.805492 22125 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:48.806157 22125 heartbeater.cc:511] Master 127.10.71.190:34151 requested a full tablet report, sending...
I20260619 14:58:48.807406 21775 ts_manager.cc:194] Registered new tserver with Master: f30e4cea167842df9a264a615a690765 (127.10.71.130:40757)
I20260619 14:58:48.808133 21775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:50761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:58:48.826097 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:48483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34151
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:58:48.942345 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:58:48.942636 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:58:48.942694 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:58:48.947785 22130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:58:48.947919 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:58:48.948040 22130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:58:48.956264 22130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.71.148:48483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:58:48.957612 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:58:48.958850 22130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
W20260619 14:58:48.972393 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:48.972781 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:58:48.977047 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:58:48.978148 22130 server_base.cc:1034] running on GCE node
I20260619 14:58:48.978603 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:58:48.979301 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:58:48.980516 22130 hybrid_clock.cc:648] HybridClock initialized: now 1781881128980467 us; error 85 us; skew 500 ppm
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:58:48.983912 22130 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:58:48.984896 22130 webserver.cc:460] Webserver started at http://127.10.71.131:43437/ using document root <none> and password file <none>
I20260619 14:58:48.985522 22130 fs_manager.cc:359] Metadata directory not provided
I20260619 14:58:48.985661 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:58:48.985904 22130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:58:48 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:58:49.002628 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "5f5acfd2bc7241bbbd7139131db49401"
format_stamp: "Formatted at 2026-06-19 14:58:49 on dist-test-slave-3qdh"
server_key: "fd9a74b6a3fe13289359576fbae9e2e4"
server_key_iv: "f4a321e1b4e4f3fe13601100365b5610"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:49.003654 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "5f5acfd2bc7241bbbd7139131db49401"
format_stamp: "Formatted at 2026-06-19 14:58:49 on dist-test-slave-3qdh"
server_key: "fd9a74b6a3fe13289359576fbae9e2e4"
server_key_iv: "f4a321e1b4e4f3fe13601100365b5610"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:49.020190 22130 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.003s sys 0.000s
I20260619 14:58:49.046578 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:49.048085 22130 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260619 14:58:49.048202 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "5f5acfd2bc7241bbbd7139131db49401"
format_stamp: "Formatted at 2026-06-19 14:58:49 on dist-test-slave-3qdh"
server_key: "fd9a74b6a3fe13289359576fbae9e2e4"
server_key_iv: "f4a321e1b4e4f3fe13601100365b5610"
server_key_version: "kuduclusterkey@0"
I20260619 14:58:49.048391 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:58:49.068393 22130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:58:49.071703 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:58:49.071902 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:58:49.072538 22130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:58:49.073462 22130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:58:49.073561 22130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:49.073647 22130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:58:49.073701 22130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:58:49.086540 22130 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:43555
I20260619 14:58:49.087481 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:58:49.088493 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 22130
I20260619 14:58:49.088608 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:58:49.088884 10526 external_mini_cluster.cc:1428] Setting key d7b05e9c89d43902b9737d4590c3c8ce
I20260619 14:58:49.089862 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:43555 every 8 connection(s)
Jun 19 14:58:49 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:58:49.105517 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.093965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:35215 (local address 127.10.71.190:34151)
0619 14:58:49.094083 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.094087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.094100 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:58:49.094684 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.094687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.094734 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.094776 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:49.096144 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.096921 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.098403 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.098724 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.098788 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.101824 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:49.101844 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:49.101847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:49.101871 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:49.103462 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:49.103998 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:49.104001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:49.104003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:49.104086 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:49.104465 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:49.104469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:49.104470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:49.104649 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:49.104709 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.105262 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.105372 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:58:49.106946 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.093072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34151 (local address 127.10.71.131:35215)
0619 14:58:49.093552 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.093603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.094474 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.095129 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:49.095148 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:49.095428 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:58:49.095998 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.096013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.097068 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.097071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.098272 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.098282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.098953 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.098956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.099130 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.099751 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:58:49.099771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:58:49.101660 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:58:49.103619 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:49.103623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:49.103635 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:49.103902 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:49.104199 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:58:49.104201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:58:49.104203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:58:49.104369 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:58:49.104777 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:58:49.104782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:58:49.105018 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 14:58:49.105841 (+ 823us) client_negotiation.cc:240] Negotiation successful
0619 14:58:49.106115 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260619 14:58:49.108244 22259 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34151
I20260619 14:58:49.108489 22259 heartbeater.cc:464] Registering TS with master...
I20260619 14:58:49.108939 22259 heartbeater.cc:511] Master 127.10.71.190:34151 requested a full tablet report, sending...
I20260619 14:58:49.109889 21775 ts_manager.cc:194] Registered new tserver with Master: 5f5acfd2bc7241bbbd7139131db49401 (127.10.71.131:43555)
I20260619 14:58:49.110603 21775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:35215
I20260619 14:58:49.115044 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:58:49.125254 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.117256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.10.71.190:34151)
0619 14:58:49.117336 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.117339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.117879 (+ 540us) server_negotiation.cc:407] Connection header received
0619 14:58:49.118042 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.118045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.118085 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.118129 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:58:49.118814 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.119260 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.120131 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.120414 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.120507 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.122087 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:58:49.122090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:58:49.122093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:58:49.122114 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:58:49.123627 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:49.124085 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:49.124089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:49.124090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:49.124165 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:58:49.124462 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:58:49.124465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:58:49.124466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:58:49.124643 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:58:49.124731 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.125034 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.125128 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:58:49.128567 21775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:58:49.132792 21775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:58:49.155294 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.142628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:43555 (local address 127.0.0.1:33804)
0619 14:58:49.145681 (+ 3053us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.145721 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.145822 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.148073 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:49.148078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:49.148097 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:49.148214 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.148219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.151890 (+ 3671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.151893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.153856 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.153866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.154876 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.154879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.154966 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.154994 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:58:49.155079 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:58:49.155172 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2909,"mutex_wait_us":26,"thread_start_us":70,"threads_started":1}
I20260619 14:58:49.155846 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.143680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:40151 (local address 127.0.0.1:51790)
0619 14:58:49.146216 (+ 2536us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.146224 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.146273 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.148611 (+ 2338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:49.148615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:49.148637 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:49.148748 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.148756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.152962 (+ 4206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.152965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.154250 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.154261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.155421 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.155424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.155520 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.155544 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:58:49.155634 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:58:49.155707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2494,"mutex_wait_us":82,"thread_start_us":74,"threads_started":1}
I20260619 14:58:49.156037 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.144267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.10.71.131:43555)
0619 14:58:49.146452 (+ 2185us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.146465 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.146483 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:58:49.146555 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.146560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.146731 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.146854 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:49.148962 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.151778 (+ 2816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.153981 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.154627 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.154710 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.154824 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.155782 (+ 958us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.155898 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2104,"thread_start_us":168,"threads_started":1}
I20260619 14:58:49.156584 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.144823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.10.71.129:40151)
0619 14:58:49.147674 (+ 2851us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.147683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.147706 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:58:49.147791 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.147796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.147958 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.148471 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:49.150428 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.152769 (+ 2341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.154402 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.155131 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.155235 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.155361 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.156303 (+ 942us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.156430 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2141,"thread_start_us":74,"threads_started":1}
I20260619 14:58:49.159075 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 1f481e8dbfbc4488ac31102b6e29afb6 (DEFAULT_TABLE table=test-table [id=cf2dec69d1004cad9cfce51663334cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:58:49.159404 21905 tablet_service.cc:1467] Processing CreateTablet for tablet 1f481e8dbfbc4488ac31102b6e29afb6 (DEFAULT_TABLE table=test-table [id=cf2dec69d1004cad9cfce51663334cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:58:49.160102 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f481e8dbfbc4488ac31102b6e29afb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:49.160410 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f481e8dbfbc4488ac31102b6e29afb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:49.171026 22279 tablet_bootstrap.cc:492] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: Bootstrap starting.
I20260619 14:58:49.173359 22279 tablet_bootstrap.cc:654] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:49.173780 22278 tablet_bootstrap.cc:492] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Bootstrap starting.
I20260619 14:58:49.173923 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.143388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40757 (local address 127.0.0.1:44844)
0619 14:58:49.146016 (+ 2628us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.146026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.146088 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.168980 (+ 22892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:49.168983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:49.169005 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:49.169112 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.169121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.170734 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.170737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.171843 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.171857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.173504 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.173508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.173605 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.173626 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:58:49.173691 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:58:49.173796 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2577,"thread_start_us":89,"threads_started":1}
I20260619 14:58:49.174700 22279 log.cc:824] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:49.175112 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.143557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.10.71.130:40757)
0619 14:58:49.168427 (+ 24870us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.168440 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.168467 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:58:49.168559 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.168564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.168748 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.169282 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:49.169446 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.170593 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.172021 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.173230 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.173326 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.173448 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.174898 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.174989 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24745,"thread_start_us":96,"threads_started":1}
I20260619 14:58:49.177299 22278 tablet_bootstrap.cc:654] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:49.177743 22059 tablet_service.cc:1467] Processing CreateTablet for tablet 1f481e8dbfbc4488ac31102b6e29afb6 (DEFAULT_TABLE table=test-table [id=cf2dec69d1004cad9cfce51663334cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:58:49.178522 22278 log.cc:824] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:49.178709 22059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f481e8dbfbc4488ac31102b6e29afb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:58:49.180536 22279 tablet_bootstrap.cc:492] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: No bootstrap required, opened a new log
I20260619 14:58:49.180716 22279 ts_tablet_manager.cc:1397] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 14:58:49.184793 22279 raft_consensus.cc:348] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.185130 22279 raft_consensus.cc:374] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:49.185237 22279 raft_consensus.cc:729] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edec1621c14bc081a1bbab18ca9db0, State: Initialized, Role: FOLLOWER
I20260619 14:58:49.185740 22279 consensus_queue.cc:260] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.186318 22282 tablet_bootstrap.cc:492] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Bootstrap starting.
I20260619 14:58:49.186798 22279 ts_tablet_manager.cc:1428] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 14:58:49.187186 21987 heartbeater.cc:503] Master 127.10.71.190:34151 was elected leader, sending a full tablet report...
I20260619 14:58:49.188711 22282 tablet_bootstrap.cc:654] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Neither blocks nor log segments found. Creating new log.
I20260619 14:58:49.190368 22282 log.cc:824] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Log is configured to *not* fsync() on all Append() calls
I20260619 14:58:49.191607 22278 tablet_bootstrap.cc:492] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: No bootstrap required, opened a new log
I20260619 14:58:49.191819 22278 ts_tablet_manager.cc:1397] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260619 14:58:49.192214 22282 tablet_bootstrap.cc:492] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: No bootstrap required, opened a new log
I20260619 14:58:49.192433 22282 ts_tablet_manager.cc:1397] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:58:49.195549 22278 raft_consensus.cc:348] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.195838 22278 raft_consensus.cc:374] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:49.195927 22278 raft_consensus.cc:729] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5acfd2bc7241bbbd7139131db49401, State: Initialized, Role: FOLLOWER
I20260619 14:58:49.195912 22282 raft_consensus.cc:348] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.196161 22282 raft_consensus.cc:374] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:58:49.196242 22282 raft_consensus.cc:729] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e4cea167842df9a264a615a690765, State: Initialized, Role: FOLLOWER
I20260619 14:58:49.196413 22278 consensus_queue.cc:260] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.196677 22282 consensus_queue.cc:260] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.197403 22282 ts_tablet_manager.cc:1428] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:58:49.197398 22278 ts_tablet_manager.cc:1428] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:58:49.197707 22259 heartbeater.cc:503] Master 127.10.71.190:34151 was elected leader, sending a full tablet report...
I20260619 14:58:49.197721 22125 heartbeater.cc:503] Master 127.10.71.190:34151 was elected leader, sending a full tablet report...
W20260619 14:58:49.210460 21988 tablet.cc:2369] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:58:49 dist-test-slave-3qdh krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881128, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
W20260619 14:58:49.287143 22126 tablet.cc:2369] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:58:49.341727 22260 tablet.cc:2369] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:58:49.345381 22285 raft_consensus.cc:482] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:58:49.345582 22285 raft_consensus.cc:504] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.346752 22285 leader_election.cc:288] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f30e4cea167842df9a264a615a690765 (127.10.71.130:40757), e2edec1621c14bc081a1bbab18ca9db0 (127.10.71.129:40151)
I20260619 14:58:49.352795 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.347090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40757 (local address 127.10.71.131:35887)
0619 14:58:49.347668 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.347683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.347784 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.348884 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:49.348887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:49.348904 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:49.349052 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.349058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.350075 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.350078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.351274 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.351284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.352399 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.352402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.352496 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.352519 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:58:49.352593 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:58:49.352655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505}
I20260619 14:58:49.353466 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.348066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:35887 (local address 127.10.71.130:40757)
0619 14:58:49.348355 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.348359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.348373 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:58:49.348423 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.348425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.348470 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.348538 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:49.349171 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.349938 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.351400 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.351926 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.352024 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.352084 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.353284 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.353352 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260619 14:58:49.354023 22060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f481e8dbfbc4488ac31102b6e29afb6" candidate_uuid: "5f5acfd2bc7241bbbd7139131db49401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e4cea167842df9a264a615a690765" is_pre_election: true
I20260619 14:58:49.354296 22060 raft_consensus.cc:2454] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5acfd2bc7241bbbd7139131db49401 in term 0.
I20260619 14:58:49.354866 22146 leader_election.cc:302] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f5acfd2bc7241bbbd7139131db49401, f30e4cea167842df9a264a615a690765; no voters:
I20260619 14:58:49.355175 22285 raft_consensus.cc:2781] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:58:49.355240 22285 raft_consensus.cc:482] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:58:49.355274 22285 raft_consensus.cc:3037] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:49.356626 22285 raft_consensus.cc:504] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.357378 22285 leader_election.cc:288] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [CANDIDATE]: Term 1 election: Requested vote from peers f30e4cea167842df9a264a615a690765 (127.10.71.130:40757), e2edec1621c14bc081a1bbab18ca9db0 (127.10.71.129:40151)
I20260619 14:58:49.357852 22060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f481e8dbfbc4488ac31102b6e29afb6" candidate_uuid: "5f5acfd2bc7241bbbd7139131db49401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e4cea167842df9a264a615a690765"
I20260619 14:58:49.357966 22060 raft_consensus.cc:3037] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:49.359102 22060 raft_consensus.cc:2454] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5acfd2bc7241bbbd7139131db49401 in term 1.
I20260619 14:58:49.359521 22146 leader_election.cc:302] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f5acfd2bc7241bbbd7139131db49401, f30e4cea167842df9a264a615a690765; no voters:
I20260619 14:58:49.359979 22285 raft_consensus.cc:2781] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:58:49.360219 22285 raft_consensus.cc:686] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 1 LEADER]: Becoming Leader. State: Replica: 5f5acfd2bc7241bbbd7139131db49401, State: Running, Role: LEADER
I20260619 14:58:49.360606 22285 consensus_queue.cc:237] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } }
I20260619 14:58:49.364029 21773 catalog_manager.cc:5495] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f5acfd2bc7241bbbd7139131db49401 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "5f5acfd2bc7241bbbd7139131db49401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5acfd2bc7241bbbd7139131db49401" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43555 } health_report { overall_health: HEALTHY } } }
I20260619 14:58:49.369344 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.348193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:35141 (local address 127.10.71.129:40151)
0619 14:58:49.348700 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.348704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.352112 (+ 3408us) server_negotiation.cc:407] Connection header received
0619 14:58:49.352162 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.352165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.352207 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.352267 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:58:49.365524 (+ 13257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.366247 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.367996 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.368633 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.368726 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.368773 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.369151 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.369226 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260619 14:58:49.369887 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.347298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:40151 (local address 127.10.71.131:35141)
0619 14:58:49.351720 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.351733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.351826 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.365217 (+ 13391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:58:49.365221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:58:49.365239 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:58:49.365365 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.365372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.366399 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.366402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.367854 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:58:49.367865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.368894 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.368897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:58:49.368994 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.369017 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:58:49.369654 (+ 637us) client_negotiation.cc:240] Negotiation successful
0619 14:58:49.369730 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4355,"thread_start_us":112,"threads_started":1}
I20260619 14:58:49.370621 21941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f481e8dbfbc4488ac31102b6e29afb6" candidate_uuid: "5f5acfd2bc7241bbbd7139131db49401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edec1621c14bc081a1bbab18ca9db0" is_pre_election: true
I20260619 14:58:49.370915 21941 raft_consensus.cc:2454] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5acfd2bc7241bbbd7139131db49401 in term 0.
I20260619 14:58:49.371246 21940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f481e8dbfbc4488ac31102b6e29afb6" candidate_uuid: "5f5acfd2bc7241bbbd7139131db49401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edec1621c14bc081a1bbab18ca9db0"
I20260619 14:58:49.371344 21940 raft_consensus.cc:3037] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:58:49.372586 21940 raft_consensus.cc:2454] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5acfd2bc7241bbbd7139131db49401 in term 1.
I20260619 14:58:49.394574 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.390098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.10.71.131:43555)
0619 14:58:49.390336 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 14:58:49.390340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:58:49.390355 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:58:49.390425 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:58:49.390428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:58:49.390482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:58:49.390576 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:58:49.390926 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.391608 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.392781 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:58:49.393095 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:58:49.393174 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:58:49.393522 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:58:49.393989 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:58:49.394111 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:58:49.394396 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:58:49.394450 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260619 14:58:49.407601 21940 raft_consensus.cc:1261] T 1f481e8dbfbc4488ac31102b6e29afb6 P e2edec1621c14bc081a1bbab18ca9db0 [term 1 FOLLOWER]: Refusing update from remote peer 5f5acfd2bc7241bbbd7139131db49401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:58:49.408545 22060 raft_consensus.cc:1261] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 1 FOLLOWER]: Refusing update from remote peer 5f5acfd2bc7241bbbd7139131db49401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:58:49.409237 22290 consensus_queue.cc:1035] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f30e4cea167842df9a264a615a690765" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:58:49.409627 22285 consensus_queue.cc:1035] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edec1621c14bc081a1bbab18ca9db0" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:58:49.438464 22293 mvcc.cc:205] Tried to move back new op lower bound from 7298585106038468608 to 7298585105864273920. Current Snapshot: MvccSnapshot[applied={T|T < 7298585106038468608}]
I20260619 14:58:49.441500 22296 mvcc.cc:205] Tried to move back new op lower bound from 7298585106038468608 to 7298585105864273920. Current Snapshot: MvccSnapshot[applied={T|T < 7298585106038468608}]
I20260619 14:58:49.451184 21773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:58:49.451349 21773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:58:49.453873 21773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594ea1da15124aa9ad12cea064fe7475: Sending DeleteTablet for 3 replicas of tablet 1f481e8dbfbc4488ac31102b6e29afb6
I20260619 14:58:49.454824 21905 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f481e8dbfbc4488ac31102b6e29afb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:58:49 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:51790
I20260619 14:58:49.455529 22059 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f481e8dbfbc4488ac31102b6e29afb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:58:49 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:44844
I20260619 14:58:49.456048 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f481e8dbfbc4488ac31102b6e29afb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:58:49 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:33804
I20260619 14:58:49.457131 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 21856
I20260619 14:58:49.459277 22304 tablet_replica.cc:330] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: stopping tablet replica
I20260619 14:58:49.459563 22304 raft_consensus.cc:2229] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:58:49.459955 22304 raft_consensus.cc:2258] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:58:49.460285 22303 tablet_replica.cc:330] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: stopping tablet replica
I20260619 14:58:49.476725 22303 raft_consensus.cc:2229] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:58:49.477059 22303 raft_consensus.cc:2258] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:58:49.478689 22304 ts_tablet_manager.cc:1905] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:58:49.481407 22303 ts_tablet_manager.cc:1905] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:58:49.483901 22303 ts_tablet_manager.cc:1918] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:58:49.483997 22303 log.cc:1196] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/1f481e8dbfbc4488ac31102b6e29afb6
I20260619 14:58:49.486408 22304 ts_tablet_manager.cc:1918] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:58:49.486498 22304 log.cc:1196] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/1f481e8dbfbc4488ac31102b6e29afb6
I20260619 14:58:49.486783 22304 ts_tablet_manager.cc:1939] T 1f481e8dbfbc4488ac31102b6e29afb6 P 5f5acfd2bc7241bbbd7139131db49401: Deleting consensus metadata
W20260619 14:58:49.488559 21759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:40151 (error 108)
I20260619 14:58:49.496239 22303 ts_tablet_manager.cc:1939] T 1f481e8dbfbc4488ac31102b6e29afb6 P f30e4cea167842df9a264a615a690765: Deleting consensus metadata
I20260619 14:58:49.496376 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 21995
I20260619 14:58:49.498122 21762 catalog_manager.cc:4841] TS 5f5acfd2bc7241bbbd7139131db49401 (127.10.71.131:43555): tablet 1f481e8dbfbc4488ac31102b6e29afb6 (table test-table [id=cf2dec69d1004cad9cfce51663334cff]) successfully deleted
I20260619 14:58:49.503743 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:58:49.503454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:40151 (local address 127.0.0.1:51804)
0619 14:58:49.503591 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.503667 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:40151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:58:49.503887 21759 catalog_manager.cc:4584] TS e2edec1621c14bc081a1bbab18ca9db0 (127.10.71.129:40151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f481e8dbfbc4488ac31102b6e29afb6: Network error: Client connection negotiation failed: client connection to 127.10.71.129:40151: connect: Connection refused (error 111)
W20260619 14:58:49.504667 22273 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:58:49.504267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:40757 (local address 127.0.0.1:44852)
0619 14:58:49.504370 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 14:58:49.504382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:58:49.504467 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:58:49.504542 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.130:40757: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:58:49.516479 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 22130
I20260619 14:58:49.523272 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 21743
I20260619 14:58:49.527480 10526 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:58:49.873086 10526 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:58:49.873224 10526 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:58:50.217147 10526 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:58:50.217 UTC [21329] LOG: received smart shutdown request
2026-06-19 14:58:50.220 UTC [21329] LOG: background worker "logical replication launcher" (PID 21338) exited with exit code 1
2026-06-19 14:58:50.220 UTC [21333] LOG: shutting down
2026-06-19 14:58:50.352 UTC [21329] LOG: database system is shut down
2026-06-19T14:58:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43062 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:58:50 dist-test-slave-3qdh krb5kdc[22314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:58:50 dist-test-slave-3qdh krb5kdc[22314](info): set up 2 sockets
Jun 19 14:58:50 dist-test-slave-3qdh krb5kdc[22314](info): commencing operation
krb5kdc: starting...
W20260619 14:58:51.436173 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:58:51 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:58:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/rangeradmin_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/rangeradmin_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/rangerlookup_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/rangerlookup_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
I20260619 14:58:51.574496 10526 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskn16cqy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/postgres -l logfile start
2026-06-19 14:58:53.687 UTC [22344] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:58:53.687 UTC [22344] LOG: listening on IPv4 address "127.10.71.148", port 48551
2026-06-19 14:58:53.690 UTC [22344] LOG: listening on Unix socket "/tmp/.s.PGSQL.48551"
2026-06-19 14:58:53.708 UTC [22347] LOG: database system was shut down at 2026-06-19 14:58:52 UTC
2026-06-19 14:58:53.711 UTC [22344] LOG: database system is ready to accept connections
I20260619 14:58:54.636098 10526 mini_postgres.cc:96] Postgres bound to 48551
2026-06-19 14:58:54.642 UTC [22355] FATAL: database "slave" does not exist
127.10.71.148:48551 - accepting connections
I20260619 14:58:54.642979 10526 mini_ranger.cc:161] Starting Ranger...
I20260619 14:58:54.657948 10526 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:58:55.049850 10526 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:58:55.049945 10526 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-admin
2026-06-19 14:58:55,325 [I] DB FLAVOR :POSTGRES
2026-06-19 14:58:55,326 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:58:55,326 [I] Checking connection..
2026-06-19 14:58:55,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:58:55,531 [I] Checking connection passed.
2026-06-19 14:58:55,531 [I] --------- Verifying version history table ---------
2026-06-19 14:58:55,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:58:55,742 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:58:55,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:58:55,940 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:58:55,940 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:58:55,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:58:56.147 UTC [22419] WARNING: there is no transaction in progress
2026-06-19 14:58:56,153 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:58:56,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:58:56,383 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:58:56,383 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:58:56,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:58:56,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:58:56,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3qdh.c.gcp-upstream.internal','N') ;"
2026-06-19 14:58:57,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:58:57,236 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:58:57,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:58:57,458 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:58:57,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:58:57,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:58:57,906 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:58:57,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:58:58.747 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.761 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.775 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.788 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.803 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.818 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.864 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.870 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.882 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.890 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.903 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.915 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.923 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.929 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.935 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:58.941 UTC [22527] WARNING: there is no transaction in progress
2026-06-19 14:58:59,486 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:58:59,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:58:59,696 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:58:59,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:58:59,940 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:58:59,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:59:00,142 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:59:00,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:59:00,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:59:00,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3qdh.c.gcp-upstream.internal';"
2026-06-19 14:59:00,767 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:59:00,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:59:00,972 [I] DB_PATCHES have already been applied
I20260619 14:59:00.977342 10526 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-admin:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskn16cqy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:59:00.977402 10526 mini_ranger.cc:193] Using host: 127.10.71.148
I20260619 14:59:00.980773 10526 mini_ranger.cc:227] Ranger admin URL: http://127.10.71.148:49369
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:59:02.016269 10526 mini_ranger.cc:160] Time spent starting Ranger: real 7.373s user 0.000s sys 0.009s
Jun 19, 2026 2:59:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:59:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:59:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:59:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:59:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:59:21.983521 10526 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/rangerkms_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/rangerkms_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.71.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.71.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.71.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/HTTP_127.10.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:59:22 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:59:22.061478 10526 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:59:22.070333 10526 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:59:22.815521 10526 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:59:22.815593 10526 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms
I20260619 14:59:22.815610 10526 mini_ranger_kms.cc:227] Using postgres at 127.10.71.148:48551
2026-06-19 14:59:22,970 [I] DB FLAVOR :POSTGRES
2026-06-19 14:59:22,970 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:59:22,970 [I] Checking connection
2026-06-19 14:59:22,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:59:23,204 [I] connection success
2026-06-19 14:59:23,204 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:59:23,204 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:59:23,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:59:23,438 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:59:23,438 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:59:23,438 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:59:23,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.71.148:48551/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:59:23,658 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:59:24.098501 10526 mini_ranger_kms.cc:310] Created kms service
I20260619 14:59:24.291050 10526 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:59:24.327708 10526 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:59:24 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:24.674579 10526 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:59:24.674710 10526 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms:/tmp/dist-test-taskn16cqy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskn16cqy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:59:24.674728 10526 mini_ranger_kms.cc:240] Using host: 127.10.71.148
I20260619 14:59:24.678547 10526 mini_ranger_kms.cc:276] Ranger KMS PID: 22707
I20260619 14:59:24.678627 10526 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.71.148:58701
Jun 19, 2026 2:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:59:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 14:59:26 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.71.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:26.724346 10526 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.663s user 0.027s sys 0.111s
I20260619 14:59:26.724470 10526 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:59:26.724545 10526 mini_ranger_kms.cc:401] 127.10.71.148:58701/kms/v1/keys
Jun 19 14:59:26 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.71.148@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
Jun 19 14:59:27 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.71.148@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
Jun 19 14:59:29 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:59:29.247282 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:41943
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:41943
--ranger_config_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:29.345374 22758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:29.345628 22758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:29.345662 22758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:29.349406 22758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:29.349457 22758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:29.349474 22758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:29.349489 22758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:29.349502 22758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:29.354251 22758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.71.148:58701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:41943
--ranger_config_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:29.355051 22758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:29.355857 22758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:29.363245 22764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:29.363343 22766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:29.363387 22758 server_base.cc:1034] running on GCE node
W20260619 14:59:29.363242 22763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:29.364068 22758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:29.365300 22758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:29.366470 22758 hybrid_clock.cc:648] HybridClock initialized: now 1781881169366445 us; error 33 us; skew 500 ppm
Jun 19 14:59:29 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:29.372503 22758 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:29.373232 22758 webserver.cc:460] Webserver started at http://127.10.71.190:42743/ using document root <none> and password file <none>
I20260619 14:59:29.373703 22758 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:29.373752 22758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:29.373946 22758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:59:29 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:29.428484 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "013c9e3f84df4cc782b72c1dead965b9"
format_stamp: "Formatted at 2026-06-19 14:59:29 on dist-test-slave-3qdh"
server_key: "489d16c06d3b2f590529e7088e42d2e5"
server_key_iv: "85ba7ee280ef634db5da29304d4cd568"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:29.429064 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "013c9e3f84df4cc782b72c1dead965b9"
format_stamp: "Formatted at 2026-06-19 14:59:29 on dist-test-slave-3qdh"
server_key: "489d16c06d3b2f590529e7088e42d2e5"
server_key_iv: "85ba7ee280ef634db5da29304d4cd568"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:29.432868 22758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:59:29.445315 22773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:29.446535 22758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:29.446663 22758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "013c9e3f84df4cc782b72c1dead965b9"
format_stamp: "Formatted at 2026-06-19 14:59:29 on dist-test-slave-3qdh"
server_key: "489d16c06d3b2f590529e7088e42d2e5"
server_key_iv: "85ba7ee280ef634db5da29304d4cd568"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:29.446787 22758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:29.477573 22758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:29.480062 22758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:29.480513 22758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:29.487824 22758 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:41943
I20260619 14:59:29.487829 22825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:41943 every 8 connection(s)
I20260619 14:59:29.488718 22758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:29.491089 22826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:29.493021 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 22758
I20260619 14:59:29.493178 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:59:29.493479 10526 external_mini_cluster.cc:1428] Setting key 62b73cea471105732f03cd22a468f8cf
I20260619 14:59:29.496779 22826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Bootstrap starting.
I20260619 14:59:29.498909 22826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:29.499753 22826 log.cc:824] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:29.501657 22826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: No bootstrap required, opened a new log
Jun 19 14:59:29 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:29.505348 22826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:29.505571 22826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:29.505658 22826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013c9e3f84df4cc782b72c1dead965b9, State: Initialized, Role: FOLLOWER
I20260619 14:59:29.506083 22826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:29.506253 22826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:29.506362 22826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:29.506449 22826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:29.507254 22826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:29.507630 22826 leader_election.cc:302] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 013c9e3f84df4cc782b72c1dead965b9; no voters:
I20260619 14:59:29.507951 22826 leader_election.cc:288] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:59:29.508013 22831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:29.508229 22831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 LEADER]: Becoming Leader. State: Replica: 013c9e3f84df4cc782b72c1dead965b9, State: Running, Role: LEADER
I20260619 14:59:29.508539 22831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:29.508625 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:29.494802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.10.71.190:41943)
0619 14:59:29.495196 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:29.495212 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:29.495249 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:59:29.496125 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:29.496146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:29.496583 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:29.496895 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:29.497695 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:29.499063 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:29.500145 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:29.500767 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:29.500912 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:29.503591 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:29.503598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:29.503613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:29.503641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:29.505398 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:29.505821 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:29.505826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:29.505831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:29.505939 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:29.506472 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:29.506475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:29.506477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:29.506919 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:29.507085 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:29.507376 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 14:59:29.507806 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260619 14:59:29.509126 22826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:29.509601 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 013c9e3f84df4cc782b72c1dead965b9. Latest consensus state: current_term: 1 leader_uuid: "013c9e3f84df4cc782b72c1dead965b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } } }
I20260619 14:59:29.509608 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "013c9e3f84df4cc782b72c1dead965b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } } }
I20260619 14:59:29.509830 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:29.509752 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:29.510658 22838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:29.511018 22826 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:59:29.516141 22838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:29.521126 22838 catalog_manager.cc:1347] Generated new cluster ID: b3e98452f6954bd89d227497b55a1c44
I20260619 14:59:29.521173 22838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:29.539381 22838 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:59:29.540550 22838 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:29.551342 22838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Generated new TSK 0
I20260619 14:59:29.551949 22838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:59:30.154665 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41943
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:30.282882 22871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:30.283138 22871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:30.283231 22871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:30.286983 22871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:30.287057 22871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:30.287163 22871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:59:30.291913 22871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.71.148:58701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:30.292807 22871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:30.293771 22871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:30.303321 22878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:30.303468 22871 server_base.cc:1034] running on GCE node
W20260619 14:59:30.303550 22879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:30.303395 22881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:30.304003 22871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:30.304677 22871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:30.305874 22871 hybrid_clock.cc:648] HybridClock initialized: now 1781881170305858 us; error 29 us; skew 500 ppm
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:30.309105 22871 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:59:30.310025 22871 webserver.cc:460] Webserver started at http://127.10.71.129:36415/ using document root <none> and password file <none>
I20260619 14:59:30.310614 22871 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:30.310694 22871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:30.310941 22871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:30.326393 22871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "d1c06b59af4d4101a2bad97f1dcf44c5"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "86e104a3de9d81cbda96c954b2ee3360"
server_key_iv: "ce32f20487e3b9590b65824161cd53af"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.327085 22871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "d1c06b59af4d4101a2bad97f1dcf44c5"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "86e104a3de9d81cbda96c954b2ee3360"
server_key_iv: "ce32f20487e3b9590b65824161cd53af"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.337617 22871 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.003s
I20260619 14:59:30.354266 22888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.355718 22871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:59:30.355826 22871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "d1c06b59af4d4101a2bad97f1dcf44c5"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "86e104a3de9d81cbda96c954b2ee3360"
server_key_iv: "ce32f20487e3b9590b65824161cd53af"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.355922 22871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:30.377310 22871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:30.380431 22871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:30.380642 22871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:30.381227 22871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:30.382092 22871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:30.382181 22871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.382308 22871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:30.382378 22871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.391646 22871 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:37499
I20260619 14:59:30.392774 22871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:59:30.393106 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 22871
I20260619 14:59:30.393245 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:59:30.393570 10526 external_mini_cluster.cc:1428] Setting key accb2e89f4b7abe1f0bce37e98c4194a
I20260619 14:59:30.402288 23001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:37499 every 8 connection(s)
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:30.421684 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:30.401289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41943 (local address 127.10.71.129:41335)
0619 14:59:30.401716 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:30.401763 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:30.402636 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:30.405043 (+ 2407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:30.405072 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:30.405402 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:30.406017 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:30.406039 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.407202 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.407206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:30.408595 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:30.408608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.409234 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.409239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:30.409545 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:30.410290 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:30.410311 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:30.416661 (+ 6350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:30.418801 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:30.418806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:30.418818 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:30.419106 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:30.419395 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:30.419398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:30.419400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:30.419560 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:30.420009 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:30.420015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:30.420281 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 14:59:30.420511 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 14:59:30.420784 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":135,"threads_started":1}
I20260619 14:59:30.422333 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:30.402208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:41335 (local address 127.10.71.190:41943)
0619 14:59:30.404650 (+ 2442us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:30.404660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:30.404676 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:59:30.404719 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:30.404724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:30.404778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:30.404907 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:30.406205 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.407050 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.408748 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.409072 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.409173 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:30.416845 (+ 7672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:30.416848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:30.416851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:30.416877 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:30.418658 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:30.419219 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:30.419223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:30.419224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:30.419306 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:30.419668 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:30.419672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:30.419673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:30.419869 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:30.419958 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:30.422059 (+ 2101us) server_negotiation.cc:299] Negotiation successful
0619 14:59:30.422175 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351,"thread_start_us":92,"threads_started":1}
I20260619 14:59:30.423933 23002 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41943
I20260619 14:59:30.424155 23002 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:30.424710 23002 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:30.426208 22790 ts_manager.cc:194] Registered new tserver with Master: d1c06b59af4d4101a2bad97f1dcf44c5 (127.10.71.129:37499)
I20260619 14:59:30.427520 22790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:41335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:59:30.450709 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41943
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:59:30.570281 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:30.570600 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:30.570685 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:30.575512 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:30.575615 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:30.575711 23010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:59:30.582041 23010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.71.148:58701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:30.583178 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:30.584337 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:30.592898 23019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:30.593025 23016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:30.592979 23017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:30.593559 23010 server_base.cc:1034] running on GCE node
I20260619 14:59:30.593919 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:30.594527 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:30.595716 23010 hybrid_clock.cc:648] HybridClock initialized: now 1781881170595695 us; error 35 us; skew 500 ppm
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:30.598225 23010 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:59:30.599020 23010 webserver.cc:460] Webserver started at http://127.10.71.130:41399/ using document root <none> and password file <none>
I20260619 14:59:30.599514 23010 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:30.599589 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:30.599781 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:30.612360 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "2bc2ab2046a2469aa93ce33f81fa8296"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "fa14adad372d5f4905a65f48bdf13121"
server_key_iv: "d2785664cce5c9f42bc0dd76427406a3"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.613070 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "2bc2ab2046a2469aa93ce33f81fa8296"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "fa14adad372d5f4905a65f48bdf13121"
server_key_iv: "d2785664cce5c9f42bc0dd76427406a3"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.618263 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 14:59:30.631455 23026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.633026 23010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:59:30.633128 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "2bc2ab2046a2469aa93ce33f81fa8296"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "fa14adad372d5f4905a65f48bdf13121"
server_key_iv: "d2785664cce5c9f42bc0dd76427406a3"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.633261 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:30.656834 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:30.663236 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:30.663458 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:30.664166 23010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:30.665174 23010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:30.665269 23010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.665359 23010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:30.665416 23010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.675056 23010 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:45683
I20260619 14:59:30.676025 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:59:30.678115 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23010
I20260619 14:59:30.678233 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:59:30.678515 10526 external_mini_cluster.cc:1428] Setting key d03e87871d0775632f8c756297db1b0b
I20260619 14:59:30.686506 23139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:45683 every 8 connection(s)
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:30.702147 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:30.688318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:43387 (local address 127.10.71.190:41943)
0619 14:59:30.688451 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:30.688457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:30.689058 (+ 601us) server_negotiation.cc:407] Connection header received
0619 14:59:30.690285 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:30.690290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:30.690347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:30.690401 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:30.691888 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.692641 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.694313 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.694723 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.694777 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:30.697971 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:30.697977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:30.697983 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:30.698012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:30.699758 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:30.700378 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:30.700382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:30.700384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:30.700462 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:30.700853 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:30.700859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:30.700863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:30.701079 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:30.701159 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:30.701801 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 14:59:30.701949 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:59:30.703768 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:30.687286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41943 (local address 127.10.71.130:43387)
0619 14:59:30.688917 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:30.688966 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:30.690055 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:30.690771 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:30.690781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:30.691094 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:30.691722 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:30.691742 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.692789 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.692795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:30.694168 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:30.694178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:30.694911 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:30.694913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:30.695026 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:30.695586 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:30.695605 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:30.697807 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:30.699883 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:30.699889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:30.699903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:30.700219 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:30.700577 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:30.700582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:30.700587 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:30.700735 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:30.701222 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:30.701230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:30.701502 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 14:59:30.702370 (+ 868us) client_negotiation.cc:240] Negotiation successful
0619 14:59:30.702702 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":115,"threads_started":1}
I20260619 14:59:30.705099 23140 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41943
I20260619 14:59:30.705317 23140 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:30.705832 23140 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:30.706903 22790 ts_manager.cc:194] Registered new tserver with Master: 2bc2ab2046a2469aa93ce33f81fa8296 (127.10.71.130:45683)
I20260619 14:59:30.707546 22790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:43387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:59:30.715613 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41943
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:30.833212 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:30.833462 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:30.833537 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:30.837918 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:30.837980 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:30.838071 23145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:59:30.844548 23145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.71.148:58701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:30.845520 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:30.846560 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:30.857954 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:30.858353 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:30.859588 23145 server_base.cc:1034] running on GCE node
W20260619 14:59:30.861023 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:30.861421 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:30.862146 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:30.863341 23145 hybrid_clock.cc:648] HybridClock initialized: now 1781881170863284 us; error 70 us; skew 500 ppm
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:30.866626 23145 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:59:30.867565 23145 webserver.cc:460] Webserver started at http://127.10.71.131:45975/ using document root <none> and password file <none>
I20260619 14:59:30.868194 23145 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:30.868273 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:30.868515 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:30.883919 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "95dde55ece5f47018cdb92eccc5c59d9"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "4324ebd6f60c4f69b3cde0577349fb88"
server_key_iv: "50e5b40419741c98d53816d3dcee098d"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.884688 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "95dde55ece5f47018cdb92eccc5c59d9"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "4324ebd6f60c4f69b3cde0577349fb88"
server_key_iv: "50e5b40419741c98d53816d3dcee098d"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.901340 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.003s sys 0.001s
I20260619 14:59:30.923640 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.925093 23145 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.001s
I20260619 14:59:30.925380 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "95dde55ece5f47018cdb92eccc5c59d9"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "4324ebd6f60c4f69b3cde0577349fb88"
server_key_iv: "50e5b40419741c98d53816d3dcee098d"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:30.925529 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 14:59:30 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:30.947150 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:30.953083 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:30.953284 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:30.953912 23145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:30.954916 23145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:30.954998 23145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.955065 23145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:30.955101 23145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:30.971365 23145 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:41805
I20260619 14:59:30.980441 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:59:30.981945 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23145
I20260619 14:59:30.982100 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:59:30.982407 10526 external_mini_cluster.cc:1428] Setting key 690ec1fcdc26654399e7ca7d5963d1a2
I20260619 14:59:30.997411 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:41805 every 8 connection(s)
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881170, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:31.022756 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:31.000416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:53947 (local address 127.10.71.190:41943)
0619 14:59:31.000518 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:31.000522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:31.001650 (+ 1128us) server_negotiation.cc:407] Connection header received
0619 14:59:31.002657 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:31.002660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:31.002706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:31.002746 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:31.004221 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.005030 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.010493 (+ 5463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.010823 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.010888 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:31.018686 (+ 7798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:31.018690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:31.018692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:31.018718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:31.020513 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:31.021091 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:31.021095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:31.021097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:31.021178 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:31.021564 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:31.021568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:31.021571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:31.021770 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:31.021848 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:31.022474 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 14:59:31.022607 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:59:31.024230 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:30.999390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41943 (local address 127.10.71.131:53947)
0619 14:59:31.001516 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:31.001573 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:31.002450 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:31.003088 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:31.003096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:31.003446 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:31.004068 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:31.004084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.006061 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.006064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:31.010379 (+ 4315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:31.010389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.011084 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.011087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:31.011250 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:31.016914 (+ 5664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:31.016932 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:31.018487 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:31.020648 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:31.020653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:31.020668 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:31.020977 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:31.021278 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:31.021281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:31.021282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:31.021442 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:31.021919 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:31.021925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:31.022210 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:59:31.022905 (+ 695us) client_negotiation.cc:240] Negotiation successful
0619 14:59:31.023208 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":144,"threads_started":1}
I20260619 14:59:31.025391 23274 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41943
I20260619 14:59:31.025641 23274 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.026157 23274 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:31.027112 22790 ts_manager.cc:194] Registered new tserver with Master: 95dde55ece5f47018cdb92eccc5c59d9 (127.10.71.131:41805)
I20260619 14:59:31.027839 22790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:53947
I20260619 14:59:31.031958 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:59:31.032626 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 22871
I20260619 14:59:31.040094 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23010
I20260619 14:59:31.045490 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23145
I20260619 14:59:31.051930 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 22758
I20260619 14:59:31.059290 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:41943
--webserver_interface=127.10.71.190
--webserver_port=42743
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:41943
--ranger_config_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:31.155318 23277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:31.155508 23277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:31.155541 23277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:31.159060 23277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:31.159097 23277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:31.159113 23277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:31.159126 23277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:31.159139 23277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:31.163595 23277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:41943
--ranger_config_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:31.164352 23277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:31.165128 23277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:31.172482 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:31.172502 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.172544 23277 server_base.cc:1034] running on GCE node
W20260619 14:59:31.172509 23283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.173108 23277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:31.174018 23277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:31.175239 23277 hybrid_clock.cc:648] HybridClock initialized: now 1781881171175230 us; error 24 us; skew 500 ppm
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:31.177798 23277 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:31.178475 23277 webserver.cc:460] Webserver started at http://127.10.71.190:42743/ using document root <none> and password file <none>
I20260619 14:59:31.178886 23277 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:31.178987 23277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:31.225602 23277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:59:31.227936 23293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.229317 23277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:31.229416 23277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "013c9e3f84df4cc782b72c1dead965b9"
format_stamp: "Formatted at 2026-06-19 14:59:29 on dist-test-slave-3qdh"
server_key: "489d16c06d3b2f590529e7088e42d2e5"
server_key_iv: "85ba7ee280ef634db5da29304d4cd568"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:31.229832 23277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:31.240931 23277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:31.243386 23277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:31.243515 23277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:31.250108 23277 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:41943
I20260619 14:59:31.250115 23345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:41943 every 8 connection(s)
I20260619 14:59:31.250993 23277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:31.254598 23346 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:59:31.255524 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23277
I20260619 14:59:31.255970 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:37499
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:41943
--webserver_port=36415
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:31.256131 23346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Bootstrap starting.
I20260619 14:59:31.271376 23346 log.cc:824] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:31.277079 23346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:59:31.277408 23346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Bootstrap complete.
I20260619 14:59:31.281108 23346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:31.281474 23346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013c9e3f84df4cc782b72c1dead965b9, State: Initialized, Role: FOLLOWER
I20260619 14:59:31.281939 23346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:31.282079 23346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:31.282137 23346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:31.282213 23346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:59:31.284248 23346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:31.284584 23346 leader_election.cc:302] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 013c9e3f84df4cc782b72c1dead965b9; no voters:
I20260619 14:59:31.284858 23346 leader_election.cc:288] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:59:31.284919 23350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:59:31.285112 23350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [term 2 LEADER]: Becoming Leader. State: Replica: 013c9e3f84df4cc782b72c1dead965b9, State: Running, Role: LEADER
I20260619 14:59:31.285346 23350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } }
I20260619 14:59:31.285866 23346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:31.286108 23351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "013c9e3f84df4cc782b72c1dead965b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } } }
I20260619 14:59:31.286221 23351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:31.286151 23352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 013c9e3f84df4cc782b72c1dead965b9. Latest consensus state: current_term: 2 leader_uuid: "013c9e3f84df4cc782b72c1dead965b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c9e3f84df4cc782b72c1dead965b9" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41943 } } }
I20260619 14:59:31.286278 23352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:31.287533 23354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:31.287889 23346 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:59:31.290278 23354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:31.290776 23354 catalog_manager.cc:1259] Loaded cluster ID: b3e98452f6954bd89d227497b55a1c44
I20260619 14:59:31.290814 23354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:31.294411 23354 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:31.295981 23354 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Loaded TSK: 0
I20260619 14:59:31.297314 23354 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:59:31.383863 23348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:31.384148 23348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:31.384207 23348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:31.389545 23348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:31.389602 23348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:31.389676 23348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:59:31.396520 23348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:37499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=36415
--tserver_master_addrs=127.10.71.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:31.397552 23348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:31.398689 23348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:31.409580 23384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.410467 23348 server_base.cc:1034] running on GCE node
W20260619 14:59:31.409582 23386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:31.412809 23383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.413211 23348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:31.413908 23348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:31.415087 23348 hybrid_clock.cc:648] HybridClock initialized: now 1781881171415049 us; error 56 us; skew 500 ppm
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:31.418298 23348 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:59:31.419152 23348 webserver.cc:460] Webserver started at http://127.10.71.129:36415/ using document root <none> and password file <none>
I20260619 14:59:31.419766 23348 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:31.419852 23348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:31.437099 23348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:31.440704 23394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.441957 23348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:59:31.442078 23348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "d1c06b59af4d4101a2bad97f1dcf44c5"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "86e104a3de9d81cbda96c954b2ee3360"
server_key_iv: "ce32f20487e3b9590b65824161cd53af"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:31.442636 23348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:31.471398 23348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:31.474597 23348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:31.474773 23348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:31.475376 23348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:31.476394 23348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:31.476449 23348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.476500 23348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:31.476526 23348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.493634 23348 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:37499
I20260619 14:59:31.493673 23507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:37499 every 8 connection(s)
I20260619 14:59:31.494602 23348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:59:31.502242 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23348
I20260619 14:59:31.502735 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:45683
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:41943
--webserver_port=41399
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:31.516598 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:31.496927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:57433 (local address 127.10.71.190:41943)
0619 14:59:31.498843 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:31.498857 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:31.498886 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:59:31.499151 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:31.499166 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:31.499479 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:31.499767 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:31.500697 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.501900 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.507286 (+ 5386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.507973 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.508126 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:31.511030 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:31.511037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:31.511051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:31.511082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:31.512947 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:31.513613 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:31.513620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:31.513625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:31.513768 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:31.514211 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:31.514215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:31.514217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:31.514666 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:31.514780 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:31.515427 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 14:59:31.515682 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1793,"thread_start_us":111,"threads_started":1}
I20260619 14:59:31.518270 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:31.497068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41943 (local address 127.10.71.129:57433)
0619 14:59:31.497832 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:31.497863 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:31.498507 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:31.499821 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:31.499827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:31.500108 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:31.500552 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:31.500565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.504392 (+ 3827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.504397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:31.505661 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:31.505674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.508394 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.508401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:31.508579 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:31.509184 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:31.509205 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:31.510815 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:31.513094 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:31.513100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:31.513110 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:31.513459 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:31.513901 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:31.513904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:31.513906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:31.514088 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:31.514848 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:31.514853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:31.515109 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 14:59:31.517152 (+ 2043us) client_negotiation.cc:240] Negotiation successful
0619 14:59:31.517480 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":91,"threads_started":1}
I20260619 14:59:31.520967 23508 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41943
I20260619 14:59:31.521205 23508 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.521675 23508 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
W20260619 14:59:31.523726 23508 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:41943 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:59:31.523846 23508 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.524022 23508 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:31.524930 23508 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.525099 23508 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
W20260619 14:59:31.525923 23508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:59:31.617764 23512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:31.617945 23512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:31.617980 23512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:31.621893 23512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:31.621954 23512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:31.622040 23512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:59:31.626729 23512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:45683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=41399
--tserver_master_addrs=127.10.71.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:31.627578 23512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:31.628597 23512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:31.636941 23517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.638003 23512 server_base.cc:1034] running on GCE node
W20260619 14:59:31.638309 23520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:31.638550 23518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.638967 23512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:31.639667 23512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:31.643047 23512 hybrid_clock.cc:648] HybridClock initialized: now 1781881171643012 us; error 50 us; skew 500 ppm
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:31.649259 23512 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:59:31.650347 23512 webserver.cc:460] Webserver started at http://127.10.71.130:41399/ using document root <none> and password file <none>
I20260619 14:59:31.651099 23512 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:31.651197 23512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:31.681303 23512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:59:31.684144 23528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.685662 23512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:31.685779 23512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "2bc2ab2046a2469aa93ce33f81fa8296"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "fa14adad372d5f4905a65f48bdf13121"
server_key_iv: "d2785664cce5c9f42bc0dd76427406a3"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:31.686282 23512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:31.710216 23512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:31.713687 23512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:31.713898 23512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:31.714459 23512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:31.715286 23512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:31.715353 23512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.715410 23512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:31.715461 23512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.725317 23512 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:45683
I20260619 14:59:31.726301 23512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:59:31.728343 23641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:45683 every 8 connection(s)
I20260619 14:59:31.732604 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23512
I20260619 14:59:31.733127 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.71.148:58701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:41805
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:41943
--webserver_port=45975
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:31.745183 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:31.729699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:36555 (local address 127.10.71.190:41943)
0619 14:59:31.729821 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:31.729826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:31.732323 (+ 2497us) server_negotiation.cc:407] Connection header received
0619 14:59:31.733801 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:31.733808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:31.733882 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:31.733946 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:31.735462 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.736238 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.737591 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.737812 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.737924 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:31.740475 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:31.740481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:31.740485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:31.740515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:31.742362 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:31.743010 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:31.743016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:31.743020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:31.743131 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:31.743630 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:31.743637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:31.743641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:31.743913 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:31.744020 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:31.744806 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 14:59:31.744945 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:59:31.746816 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:31.730072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41943 (local address 127.10.71.130:36555)
0619 14:59:31.732166 (+ 2094us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:31.732204 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:31.733470 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:31.734360 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:31.734372 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:31.734778 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:31.735325 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:31.735339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.736439 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.736443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:31.737456 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:31.737466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:31.737967 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:31.737971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:31.738197 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:31.738841 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:31.738864 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:31.740278 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:31.742506 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:31.742513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:31.742528 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:31.742866 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:31.743274 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:31.743279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:31.743282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:31.743488 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:31.744125 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:31.744132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:31.744498 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 14:59:31.745550 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0619 14:59:31.745861 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1935,"thread_start_us":95,"threads_started":1}
I20260619 14:59:31.748180 23642 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41943
I20260619 14:59:31.748425 23642 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.748886 23642 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
W20260619 14:59:31.750422 23642 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:41943 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:59:31.750576 23642 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.750753 23642 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:31.751543 23642 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:31.751708 23642 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
W20260619 14:59:31.752525 23642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:59:31.871042 23645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:31.871315 23645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:31.871376 23645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:31.876776 23645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:31.876838 23645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:31.876940 23645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:59:31.883136 23645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.71.148:58701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:41805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=45975
--tserver_master_addrs=127.10.71.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:31.884145 23645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:31.885074 23645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:31.894338 23651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:31.894456 23654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:31.894328 23652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:31.894923 23645 server_base.cc:1034] running on GCE node
I20260619 14:59:31.895279 23645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:31.895886 23645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:31.899173 23645 hybrid_clock.cc:648] HybridClock initialized: now 1781881171899142 us; error 39 us; skew 500 ppm
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:31.902246 23645 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:59:31.903141 23645 webserver.cc:460] Webserver started at http://127.10.71.131:45975/ using document root <none> and password file <none>
I20260619 14:59:31.903721 23645 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:31.903847 23645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:59:31 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:31.923851 23645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:59:31.927419 23662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.928915 23645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:59:31.929037 23645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "95dde55ece5f47018cdb92eccc5c59d9"
format_stamp: "Formatted at 2026-06-19 14:59:30 on dist-test-slave-3qdh"
server_key: "4324ebd6f60c4f69b3cde0577349fb88"
server_key_iv: "50e5b40419741c98d53816d3dcee098d"
server_key_version: "kuduclusterkey@0"
I20260619 14:59:31.929615 23645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:31.984479 23645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:31.987670 23645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:31.987907 23645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:31.988582 23645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:31.989444 23645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:31.989511 23645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:31.989575 23645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:31.989642 23645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:32.000597 23645 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:41805
I20260619 14:59:32.000679 23782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:41805 every 8 connection(s)
I20260619 14:59:32.001662 23645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:59:32.010529 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23645
Jun 19 14:59:32 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881171, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:32.015827 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.003512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:60667 (local address 127.10.71.190:41943)
0619 14:59:32.003659 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.003665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.004632 (+ 967us) server_negotiation.cc:407] Connection header received
0619 14:59:32.005729 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.005732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.005776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.005868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:32.007056 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.007525 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.008976 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.009163 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.009239 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.011752 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:32.011757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:32.011759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:32.011783 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:32.013491 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:32.014066 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:32.014070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:32.014073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:32.014168 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:32.014581 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:32.014587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:32.014592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:32.014834 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:32.014943 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.015498 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.015632 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:59:32.016633 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.003951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41943 (local address 127.10.71.131:60667)
0619 14:59:32.004475 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.004529 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.005522 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.006043 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.006053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.006342 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:32.006871 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.006886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.007643 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.007646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.008873 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.008883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.009298 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.009301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.009456 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.009946 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:32.009963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:32.011598 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:32.013617 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:32.013623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:32.013636 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:32.013950 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:32.014290 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:32.014294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:32.014296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:32.014460 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:32.014963 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:32.014968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:32.015230 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.015429 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.015693 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260619 14:59:32.017935 23783 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41943
I20260619 14:59:32.018152 23783 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:32.018589 23783 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:32.019814 23310 ts_manager.cc:194] Registered new tserver with Master: 95dde55ece5f47018cdb92eccc5c59d9 (127.10.71.131:41805)
I20260619 14:59:32.021178 23310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:60667
I20260619 14:59:32.022084 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.011189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.10.71.190:41943)
0619 14:59:32.011740 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.011748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.011759 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:59:32.011785 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.011817 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.011867 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.011957 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:32.012832 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.013631 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.014682 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.015045 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.015148 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.016875 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:32.016879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:32.016882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:32.016906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:32.019688 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:32.020197 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:32.020201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:32.020203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:32.020292 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:32.020637 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:32.020641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:32.020643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:32.020848 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:32.021069 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.021789 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.021900 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":101,"threads_started":1}
Jun 19 14:59:32 dist-test-slave-3qdh krb5kdc[22314](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781881172, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:32.526119 23508 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:32.526377 23508 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:32.527064 23310 ts_manager.cc:194] Registered new tserver with Master: d1c06b59af4d4101a2bad97f1dcf44c5 (127.10.71.129:37499)
I20260619 14:59:32.527798 23310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:57433
Jun 19 14:59:32 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881172, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:32.752720 23642 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:32.752955 23642 heartbeater.cc:511] Master 127.10.71.190:41943 requested a full tablet report, sending...
I20260619 14:59:32.753659 23310 ts_manager.cc:194] Registered new tserver with Master: 2bc2ab2046a2469aa93ce33f81fa8296 (127.10.71.130:45683)
I20260619 14:59:32.754410 23310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:36555
I20260619 14:59:32.763684 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:59:32.774231 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.765321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.10.71.190:41943)
0619 14:59:32.765519 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.765523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.765728 (+ 205us) server_negotiation.cc:407] Connection header received
0619 14:59:32.765838 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.765840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.765881 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.765963 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:32.766867 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.767686 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.768938 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.769270 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.769370 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.770963 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:32.770968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:32.770973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:32.771001 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:32.772531 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:32.772993 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:32.772996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:32.772998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:32.773080 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:32.773355 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:32.773357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:32.773359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:32.773569 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:32.773694 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.773958 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.774074 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":56,"threads_started":1}
I20260619 14:59:32.777571 23310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:59:32.781522 23310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:59:32.799300 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.791975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37499 (local address 127.0.0.1:38352)
0619 14:59:32.793592 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.793605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.793683 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.794785 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.794790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.794809 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.794912 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.794917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.796565 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.796568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.797744 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.797753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.798852 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.798855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.798954 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.798992 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.799073 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.799179 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"thread_start_us":81,"threads_started":1}
I20260619 14:59:32.800055 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.793148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.10.71.129:37499)
0619 14:59:32.794199 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.794205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.794226 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:59:32.794298 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.794302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.794486 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.794650 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.795035 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.796433 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.797865 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.798573 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.798659 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.798795 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.799832 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.799924 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":73,"threads_started":1}
I20260619 14:59:32.801187 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.791758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:41805 (local address 127.0.0.1:37080)
0619 14:59:32.793189 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.793229 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.793337 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.796956 (+ 3619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.796959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.796972 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.797048 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.797053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.798144 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.798147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.799688 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.799698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.800852 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.800855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.800947 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.800967 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.801035 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.801085 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":71,"threads_started":1}
I20260619 14:59:32.801676 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.792564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.10.71.131:41805)
0619 14:59:32.796488 (+ 3924us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.796494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.796521 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:59:32.796583 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.796587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.796736 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.796847 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.797150 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.798035 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.799833 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.800586 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.800679 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.800796 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.801477 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.801565 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3836,"thread_start_us":103,"threads_started":1}
I20260619 14:59:32.802613 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.792570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.10.71.130:45683)
0619 14:59:32.794046 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.794053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.798438 (+ 4385us) server_negotiation.cc:407] Connection header received
0619 14:59:32.798503 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.798507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.798661 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.798770 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.799119 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.800123 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.801434 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.802064 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.802157 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.802262 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.802394 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.802490 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":74,"threads_started":1}
I20260619 14:59:32.802922 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.792865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:45683 (local address 127.0.0.1:48198)
0619 14:59:32.798265 (+ 5400us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.798277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.798360 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.798896 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.798899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.798913 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.798988 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.798993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.800247 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.800250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.801314 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.801324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.802176 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.802179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.802256 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.802276 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.802753 (+ 477us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.802815 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5346,"thread_start_us":71,"threads_started":1}
I20260619 14:59:32.802939 23433 tablet_service.cc:1467] Processing CreateTablet for tablet 6c21fd246e4141db8f70208d2bbadae8 (DEFAULT_TABLE table=test-table [id=51bf4039da3045df907b8cb1e02f6c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:32.804011 23433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c21fd246e4141db8f70208d2bbadae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:32.806767 23717 tablet_service.cc:1467] Processing CreateTablet for tablet 6c21fd246e4141db8f70208d2bbadae8 (DEFAULT_TABLE table=test-table [id=51bf4039da3045df907b8cb1e02f6c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:32.807582 23717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c21fd246e4141db8f70208d2bbadae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:32.807406 23568 tablet_service.cc:1467] Processing CreateTablet for tablet 6c21fd246e4141db8f70208d2bbadae8 (DEFAULT_TABLE table=test-table [id=51bf4039da3045df907b8cb1e02f6c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:32.808267 23568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c21fd246e4141db8f70208d2bbadae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:32.820534 23810 tablet_bootstrap.cc:492] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5: Bootstrap starting.
I20260619 14:59:32.822435 23810 tablet_bootstrap.cc:654] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:32.823187 23810 log.cc:824] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:32.825129 23810 tablet_bootstrap.cc:492] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5: No bootstrap required, opened a new log
I20260619 14:59:32.825295 23810 ts_tablet_manager.cc:1397] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:59:32.828589 23809 tablet_bootstrap.cc:492] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Bootstrap starting.
I20260619 14:59:32.829164 23810 raft_consensus.cc:348] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.829419 23810 raft_consensus.cc:374] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:32.829470 23810 raft_consensus.cc:729] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c06b59af4d4101a2bad97f1dcf44c5, State: Initialized, Role: FOLLOWER
I20260619 14:59:32.829844 23810 consensus_queue.cc:260] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.830494 23809 tablet_bootstrap.cc:654] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:32.830716 23810 ts_tablet_manager.cc:1428] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:59:32.831015 23508 heartbeater.cc:503] Master 127.10.71.190:41943 was elected leader, sending a full tablet report...
I20260619 14:59:32.831200 23812 tablet_bootstrap.cc:492] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Bootstrap starting.
I20260619 14:59:32.831193 23809 log.cc:824] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:32.833142 23812 tablet_bootstrap.cc:654] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:32.833957 23812 log.cc:824] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:32.836099 23812 tablet_bootstrap.cc:492] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: No bootstrap required, opened a new log
I20260619 14:59:32.836249 23812 ts_tablet_manager.cc:1397] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:59:32.839391 23812 raft_consensus.cc:348] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.839633 23812 raft_consensus.cc:374] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:32.839672 23812 raft_consensus.cc:729] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc2ab2046a2469aa93ce33f81fa8296, State: Initialized, Role: FOLLOWER
I20260619 14:59:32.840094 23812 consensus_queue.cc:260] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.840955 23812 ts_tablet_manager.cc:1428] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 14:59:32.841163 23642 heartbeater.cc:503] Master 127.10.71.190:41943 was elected leader, sending a full tablet report...
I20260619 14:59:32.841355 23809 tablet_bootstrap.cc:492] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: No bootstrap required, opened a new log
I20260619 14:59:32.841517 23809 ts_tablet_manager.cc:1397] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260619 14:59:32.844925 23809 raft_consensus.cc:348] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.845139 23809 raft_consensus.cc:374] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:32.845184 23809 raft_consensus.cc:729] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dde55ece5f47018cdb92eccc5c59d9, State: Initialized, Role: FOLLOWER
I20260619 14:59:32.845546 23809 consensus_queue.cc:260] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.846354 23809 ts_tablet_manager.cc:1428] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260619 14:59:32.846647 23783 heartbeater.cc:503] Master 127.10.71.190:41943 was elected leader, sending a full tablet report...
Jun 19 14:59:32 dist-test-slave-3qdh krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881172, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for HTTP/127.10.71.148@KRBTEST.COM
I20260619 14:59:32.915843 23818 raft_consensus.cc:482] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:32.916057 23818 raft_consensus.cc:504] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.917515 23818 leader_election.cc:288] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bc2ab2046a2469aa93ce33f81fa8296 (127.10.71.130:45683), d1c06b59af4d4101a2bad97f1dcf44c5 (127.10.71.129:37499)
I20260619 14:59:32.927047 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.918992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:48689 (local address 127.10.71.129:37499)
0619 14:59:32.919237 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.919241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.920104 (+ 863us) server_negotiation.cc:407] Connection header received
0619 14:59:32.920663 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.920667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.920718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.920779 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.923319 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.924086 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.925767 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.926370 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.926459 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.926522 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.926849 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.926920 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260619 14:59:32.927448 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.918360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37499 (local address 127.10.71.131:48689)
0619 14:59:32.920001 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.920018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.920385 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.923043 (+ 2658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.923045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.923079 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.923200 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.923208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.924227 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.924230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.925622 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.925633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.926617 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.926620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.926718 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.926740 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.927228 (+ 488us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.927301 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":121,"threads_started":1}
I20260619 14:59:32.928344 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c21fd246e4141db8f70208d2bbadae8" candidate_uuid: "95dde55ece5f47018cdb92eccc5c59d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" is_pre_election: true
I20260619 14:59:32.928678 23462 raft_consensus.cc:2454] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95dde55ece5f47018cdb92eccc5c59d9 in term 0.
I20260619 14:59:32.929189 23664 leader_election.cc:302] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95dde55ece5f47018cdb92eccc5c59d9, d1c06b59af4d4101a2bad97f1dcf44c5; no voters:
I20260619 14:59:32.929533 23818 raft_consensus.cc:2781] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:59:32.929610 23818 raft_consensus.cc:482] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:59:32.929654 23818 raft_consensus.cc:3037] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:32.930948 23818 raft_consensus.cc:504] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.931298 23818 leader_election.cc:288] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [CANDIDATE]: Term 1 election: Requested vote from peers 2bc2ab2046a2469aa93ce33f81fa8296 (127.10.71.130:45683), d1c06b59af4d4101a2bad97f1dcf44c5 (127.10.71.129:37499)
I20260619 14:59:32.931703 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c21fd246e4141db8f70208d2bbadae8" candidate_uuid: "95dde55ece5f47018cdb92eccc5c59d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5"
I20260619 14:59:32.931815 23462 raft_consensus.cc:3037] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:32.933038 23462 raft_consensus.cc:2454] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95dde55ece5f47018cdb92eccc5c59d9 in term 1.
I20260619 14:59:32.933362 23664 leader_election.cc:302] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95dde55ece5f47018cdb92eccc5c59d9, d1c06b59af4d4101a2bad97f1dcf44c5; no voters:
I20260619 14:59:32.933568 23818 raft_consensus.cc:2781] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:32.933764 23818 raft_consensus.cc:686] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 1 LEADER]: Becoming Leader. State: Replica: 95dde55ece5f47018cdb92eccc5c59d9, State: Running, Role: LEADER
I20260619 14:59:32.934132 23818 consensus_queue.cc:237] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.937347 23309 catalog_manager.cc:5495] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95dde55ece5f47018cdb92eccc5c59d9 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "95dde55ece5f47018cdb92eccc5c59d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } health_report { overall_health: UNKNOWN } } }
I20260619 14:59:32.940476 23817 raft_consensus.cc:482] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:32.940656 23817 raft_consensus.cc:504] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dde55ece5f47018cdb92eccc5c59d9" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 41805 } } peers { permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 } } peers { permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 } }
I20260619 14:59:32.941845 23817 leader_election.cc:288] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95dde55ece5f47018cdb92eccc5c59d9 (127.10.71.131:41805), d1c06b59af4d4101a2bad97f1dcf44c5 (127.10.71.129:37499)
I20260619 14:59:32.945287 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.918838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:52987 (local address 127.10.71.130:45683)
0619 14:59:32.919134 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.919139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.919245 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:59:32.919437 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.919440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.919482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.919536 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.942030 (+ 22494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.942750 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.944170 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.944745 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.944818 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.944859 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.945133 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.945194 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260619 14:59:32.945626 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.917900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:45683 (local address 127.10.71.131:52987)
0619 14:59:32.919154 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.919168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.919315 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.919675 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.919678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.919693 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.919812 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.919817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.942880 (+ 23063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.942884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.944038 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.944047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.944937 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.944940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.945023 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.945041 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.945442 (+ 401us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.945514 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"thread_start_us":122,"threads_started":1}
I20260619 14:59:32.946234 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c21fd246e4141db8f70208d2bbadae8" candidate_uuid: "95dde55ece5f47018cdb92eccc5c59d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" is_pre_election: true
I20260619 14:59:32.946439 23583 raft_consensus.cc:2454] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95dde55ece5f47018cdb92eccc5c59d9 in term 0.
I20260619 14:59:32.946723 23582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c21fd246e4141db8f70208d2bbadae8" candidate_uuid: "95dde55ece5f47018cdb92eccc5c59d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc2ab2046a2469aa93ce33f81fa8296"
I20260619 14:59:32.946820 23582 raft_consensus.cc:3037] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:32.948037 23582 raft_consensus.cc:2454] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95dde55ece5f47018cdb92eccc5c59d9 in term 1.
I20260619 14:59:32.949044 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.942098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:37917 (local address 127.10.71.129:37499)
0619 14:59:32.942207 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.942210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.942519 (+ 309us) server_negotiation.cc:407] Connection header received
0619 14:59:32.942699 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.942702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.942741 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.942782 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.943181 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.944964 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.947571 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.948159 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.948246 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.948286 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.948871 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.948945 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:59:32.949482 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.942078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37499 (local address 127.10.71.130:37917)
0619 14:59:32.942429 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.942443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.942589 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.942895 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.942898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.942914 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.943043 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.943050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.945095 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.945098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.947432 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.947443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.948639 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.948645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.948745 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.948765 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.949287 (+ 522us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.949353 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":90,"threads_started":1}
I20260619 14:59:32.949914 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c21fd246e4141db8f70208d2bbadae8" candidate_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" is_pre_election: true
I20260619 14:59:32.950068 23462 raft_consensus.cc:2379] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2bc2ab2046a2469aa93ce33f81fa8296 in current term 1: Already voted for candidate 95dde55ece5f47018cdb92eccc5c59d9 in this term.
I20260619 14:59:32.954213 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.942475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:43173 (local address 127.10.71.131:41805)
0619 14:59:32.942575 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.942579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.950738 (+ 8159us) server_negotiation.cc:407] Connection header received
0619 14:59:32.950882 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.950884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.950921 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.950959 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:32.951260 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.951935 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.953116 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.953645 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.953726 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.953766 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.954052 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.954113 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:59:32.954721 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.942163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:41805 (local address 127.10.71.130:43173)
0619 14:59:32.950676 (+ 8513us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:32.950689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:32.950789 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:32.951061 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:32.951064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:32.951077 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:32.951139 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.951143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.952057 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.952059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.952992 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:32.953002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.953850 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.953853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:32.953928 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.953948 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:59:32.954490 (+ 542us) client_negotiation.cc:240] Negotiation successful
0619 14:59:32.954609 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8464,"thread_start_us":108,"threads_started":1}
I20260619 14:59:32.955232 23737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c21fd246e4141db8f70208d2bbadae8" candidate_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95dde55ece5f47018cdb92eccc5c59d9" is_pre_election: true
I20260619 14:59:32.955857 23529 leader_election.cc:302] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bc2ab2046a2469aa93ce33f81fa8296; no voters: 95dde55ece5f47018cdb92eccc5c59d9, d1c06b59af4d4101a2bad97f1dcf44c5
I20260619 14:59:32.956213 23817 raft_consensus.cc:2726] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 14:59:32.965202 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:32.960928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.10.71.131:41805)
0619 14:59:32.961027 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:32.961031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:32.961155 (+ 124us) server_negotiation.cc:407] Connection header received
0619 14:59:32.961290 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:32.961293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:32.961342 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:32.961424 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:59:32.961796 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.962243 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.963139 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:32.963446 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:32.963558 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:32.963957 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:59:32.964506 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:59:32.964631 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:32.965053 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 14:59:32.965097 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:59:32.971458 23462 raft_consensus.cc:1261] T 6c21fd246e4141db8f70208d2bbadae8 P d1c06b59af4d4101a2bad97f1dcf44c5 [term 1 FOLLOWER]: Refusing update from remote peer 95dde55ece5f47018cdb92eccc5c59d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:32.971483 23582 raft_consensus.cc:1261] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 1 FOLLOWER]: Refusing update from remote peer 95dde55ece5f47018cdb92eccc5c59d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:32.972117 23822 consensus_queue.cc:1035] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c06b59af4d4101a2bad97f1dcf44c5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:32.972853 23822 consensus_queue.cc:1035] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc2ab2046a2469aa93ce33f81fa8296" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 45683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:59:32.980407 23643 tablet.cc:2369] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:32.983868 23827 mvcc.cc:205] Tried to move back new op lower bound from 7298585284485308416 to 7298585284341264384. Current Snapshot: MvccSnapshot[applied={T|T < 7298585284485308416}]
I20260619 14:59:32.985283 23828 mvcc.cc:205] Tried to move back new op lower bound from 7298585284485308416 to 7298585284341264384. Current Snapshot: MvccSnapshot[applied={T|T < 7298585284485308416}]
I20260619 14:59:32.988520 23830 mvcc.cc:205] Tried to move back new op lower bound from 7298585284485308416 to 7298585284341264384. Current Snapshot: MvccSnapshot[applied={T|T < 7298585284485308416}]
I20260619 14:59:32.994956 23309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:32.995127 23309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:32.997411 23309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 013c9e3f84df4cc782b72c1dead965b9: Sending DeleteTablet for 3 replicas of tablet 6c21fd246e4141db8f70208d2bbadae8
I20260619 14:59:32.998241 23717 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c21fd246e4141db8f70208d2bbadae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:32 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:37080
I20260619 14:59:32.998780 23568 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c21fd246e4141db8f70208d2bbadae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:32 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:48198
I20260619 14:59:32.999442 23433 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c21fd246e4141db8f70208d2bbadae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:32 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:38352
I20260619 14:59:33.000031 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23348
I20260619 14:59:33.022930 23836 tablet_replica.cc:330] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: stopping tablet replica
I20260619 14:59:33.023191 23836 raft_consensus.cc:2229] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:59:33.023571 23836 raft_consensus.cc:2258] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:33.024035 23837 tablet_replica.cc:330] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: stopping tablet replica
I20260619 14:59:33.024312 23837 raft_consensus.cc:2229] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:33.024598 23837 raft_consensus.cc:2258] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:33.025846 23836 ts_tablet_manager.cc:1905] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:33.029412 23837 ts_tablet_manager.cc:1905] T 6c21fd246e4141db8f70208d2bbadae8 P 2bc2ab2046a2469aa93ce33f81fa8296: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:59:33.031817 23295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:37499 (error 108)
I20260619 14:59:33.032684 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23512
I20260619 14:59:33.028272 23836 ts_tablet_manager.cc:1918] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:33.039857 23836 log.cc:1196] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/6c21fd246e4141db8f70208d2bbadae8
I20260619 14:59:33.039929 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:33.039683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37499 (local address 127.0.0.1:38354)
0619 14:59:33.039780 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:33.039842 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:59:33.040153 23836 ts_tablet_manager.cc:1939] T 6c21fd246e4141db8f70208d2bbadae8 P 95dde55ece5f47018cdb92eccc5c59d9: Deleting consensus metadata
W20260619 14:59:33.040226 23295 catalog_manager.cc:4584] TS d1c06b59af4d4101a2bad97f1dcf44c5 (127.10.71.129:37499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c21fd246e4141db8f70208d2bbadae8: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37499: connect: Connection refused (error 111)
I20260619 14:59:33.041219 23294 catalog_manager.cc:4841] TS 95dde55ece5f47018cdb92eccc5c59d9 (127.10.71.131:41805): tablet 6c21fd246e4141db8f70208d2bbadae8 (table test-table [id=51bf4039da3045df907b8cb1e02f6c15]) successfully deleted
W20260619 14:59:33.042488 23804 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:59:33.042036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:45683 (local address 127.0.0.1:48200)
0619 14:59:33.042163 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:33.042174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:33.042253 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:33.042421 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.130:45683: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:59:33.042949 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23645
I20260619 14:59:33.049973 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23277
I20260619 14:59:33.055670 10526 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:59:33.389529 10526 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:59:33.389619 10526 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:59:33.723788 10526 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:59:33.723 UTC [22344] LOG: received smart shutdown request
2026-06-19 14:59:33.727 UTC [22344] LOG: background worker "logical replication launcher" (PID 22353) exited with exit code 1
2026-06-19 14:59:33.727 UTC [22348] LOG: shutting down
2026-06-19 14:59:33.868 UTC [22344] LOG: database system is shut down
2026-06-19T14:59:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (43527 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:59:34 dist-test-slave-3qdh krb5kdc[23848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:59:34 dist-test-slave-3qdh krb5kdc[23848](info): set up 2 sockets
Jun 19 14:59:34 dist-test-slave-3qdh krb5kdc[23848](info): commencing operation
krb5kdc: starting...
W20260619 14:59:34.931967 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:59:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:59:35.037266 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34389
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:34389,127.10.71.189:40263,127.10.71.188:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:35.133912 23861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:35.134127 23861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:35.134202 23861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:35.137712 23861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:35.137758 23861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:35.137776 23861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:35.137815 23861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:35.137849 23861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:35.142367 23861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:34389,127.10.71.189:40263,127.10.71.188:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:35.143172 23861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:35.144074 23861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:35.151425 23869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:35.151429 23867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:35.151436 23866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:35.151470 23861 server_base.cc:1034] running on GCE node
I20260619 14:59:35.152040 23861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:35.152927 23861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:35.154153 23861 hybrid_clock.cc:648] HybridClock initialized: now 1781881175154124 us; error 24 us; skew 500 ppm
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:35.161510 23861 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:35.162261 23861 webserver.cc:460] Webserver started at http://127.10.71.190:36563/ using document root <none> and password file <none>
I20260619 14:59:35.162662 23861 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:35.162710 23861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:35.162838 23861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:35.164537 23861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "574601dab7ea4ad8bdb1613510da7c06"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "6bf30d06c64c97c905b32cc83353669c"
server_key_iv: "d80bfc3924d27671003030a555c00f0e"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.165190 23861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "574601dab7ea4ad8bdb1613510da7c06"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "6bf30d06c64c97c905b32cc83353669c"
server_key_iv: "d80bfc3924d27671003030a555c00f0e"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.168232 23861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:35.170390 23876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:35.171459 23861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:35.171582 23861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "574601dab7ea4ad8bdb1613510da7c06"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "6bf30d06c64c97c905b32cc83353669c"
server_key_iv: "d80bfc3924d27671003030a555c00f0e"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.171677 23861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:35.214558 23861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:35.217217 23861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:35.217396 23861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:35.223861 23861 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:34389
I20260619 14:59:35.223887 23928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:34389 every 8 connection(s)
I20260619 14:59:35.224782 23861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:35.227264 23929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:35.230751 23929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.232127 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23861
I20260619 14:59:35.232260 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:59:35.232578 10526 external_mini_cluster.cc:1428] Setting key 41d9272cec66bde32f9906e219794cb6
WARNING: no policy specified for kudu/127.10.71.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.189@KRBTEST.COM" created.
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:35.246752 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.232847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34389 (local address 127.0.0.1:54602)
0619 14:59:35.233408 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.233439 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.234247 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.234917 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.234922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.235296 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.235519 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.235531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.236643 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.236646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.237256 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.237264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.237872 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.237876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.237986 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.238521 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.238536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.242122 (+ 3586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.244021 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.244029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.244037 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.244256 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.244707 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.244709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.244711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.244822 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.245610 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.245614 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.245718 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.245829 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.246018 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"spinlock_wait_cycles":8576,"thread_start_us":135,"threads_started":1}
I20260619 14:59:35.246762 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.232847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.10.71.190:34389)
0619 14:59:35.233224 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.233239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.233558 (+ 319us) server_negotiation.cc:407] Connection header received
0619 14:59:35.234546 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.234551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.234719 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.234917 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.235659 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.236532 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.237382 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.237724 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.237891 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.242249 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.242254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.242271 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.242289 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.243860 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.244433 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.244442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.244447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.244595 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.244944 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.244950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.244954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.245483 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.245622 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.245973 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.246143 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260619 14:59:35.248795 23929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.249380 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.249178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54506)
0619 14:59:35.249271 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.249331 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:59:35.249534 23880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
I20260619 14:59:35.250219 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.249984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54516)
0619 14:59:35.250129 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.250169 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 14:59:35.250342 23929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:40263: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab.
Entry for principal kudu/127.10.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab.
I20260619 14:59:35.256888 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:40263
--webserver_interface=127.10.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab
--principal=kudu/127.10.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:34389,127.10.71.189:40263,127.10.71.188:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:35.299530 23929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } attempt: 1
I20260619 14:59:35.300218 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.299981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54532)
0619 14:59:35.300114 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.300162 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:59:35.300863 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.300670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54546)
0619 14:59:35.300758 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.300819 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:59:35.300980 23929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:40263: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
W20260619 14:59:35.354449 23936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:35.354697 23936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:35.354795 23936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:35.358364 23936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:35.358435 23936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:35.358469 23936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:35.358501 23936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:35.358527 23936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:35.362803 23936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:34389,127.10.71.189:40263,127.10.71.188:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab
--principal=kudu/127.10.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:35.363621 23936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:35.364501 23936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:59:35.369131 23929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } attempt: 2
I20260619 14:59:35.369825 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.369586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54556)
0619 14:59:35.369699 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.369748 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:59:35.370497 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.370294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54572)
0619 14:59:35.370399 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.370448 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:59:35.370620 23929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:40263: Network error: Client connection negotiation failed: client connection to 127.10.71.189:40263: connect: Connection refused (error 111)
W20260619 14:59:35.372298 23942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:35.372378 23944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:35.372335 23941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:35.372464 23936 server_base.cc:1034] running on GCE node
I20260619 14:59:35.372924 23936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:35.373835 23936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:35.375082 23936 hybrid_clock.cc:648] HybridClock initialized: now 1781881175375052 us; error 58 us; skew 500 ppm
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:35.377509 23936 init.cc:377] Logged in from keytab as kudu/127.10.71.189@KRBTEST.COM (short username kudu)
I20260619 14:59:35.378254 23936 webserver.cc:460] Webserver started at http://127.10.71.189:43101/ using document root <none> and password file <none>
I20260619 14:59:35.378710 23936 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:35.378789 23936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:35.378988 23936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:35.380733 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data/instance:
uuid: "4a638e79c7bb426da569b74148bbb42d"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "0d10b1e5c56c0aa596f821a18ddf643b"
server_key_iv: "877736ce51210c8ee2a487c95707017e"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.381203 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal/instance:
uuid: "4a638e79c7bb426da569b74148bbb42d"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "0d10b1e5c56c0aa596f821a18ddf643b"
server_key_iv: "877736ce51210c8ee2a487c95707017e"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.384217 23936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:35.386240 23951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:35.387228 23936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:35.387382 23936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal
uuid: "4a638e79c7bb426da569b74148bbb42d"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "0d10b1e5c56c0aa596f821a18ddf643b"
server_key_iv: "877736ce51210c8ee2a487c95707017e"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.387478 23936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:35.403821 23936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:35.406219 23936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:35.406387 23936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:35.412954 23936 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.189:40263
I20260619 14:59:35.412978 24003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.189:40263 every 8 connection(s)
I20260619 14:59:35.413729 23936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
I20260619 14:59:35.416205 24004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:35.419448 24004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.421474 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 23936
I20260619 14:59:35.421618 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-1/wal/instance
I20260619 14:59:35.421895 10526 external_mini_cluster.cc:1428] Setting key 273a9bcfef46208fbcd20b8ba7f54e11
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.188@KRBTEST.COM" created.
I20260619 14:59:35.433081 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.421081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.10.71.190:34389)
0619 14:59:35.421196 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.421199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.422121 (+ 922us) server_negotiation.cc:407] Connection header received
0619 14:59:35.422961 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.422966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.423027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.423113 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.424562 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.425068 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.426720 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.426946 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.427027 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.429762 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.429766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.429767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.429787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.431150 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.431582 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.431586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.431588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.431658 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.432006 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.432010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.432016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.432183 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.432278 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.432827 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.432933 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:59:35.433702 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.421466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34389 (local address 127.0.0.1:54612)
0619 14:59:35.421971 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.422028 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.422780 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.423281 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.423291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.423753 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.424415 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.424435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.425183 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.425188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.426580 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.426594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.427091 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.427099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.427329 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.427985 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.428007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.429612 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.431260 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.431265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.431276 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.431489 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.431766 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.431769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.431771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.431899 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.432278 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.432283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.432558 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.432771 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.433044 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":167,"threads_started":1}
I20260619 14:59:35.435209 24004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 14:59:35.445616 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.435640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54588)
0619 14:59:35.435740 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.435751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.435819 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.436428 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.436430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.436540 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.436612 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.436616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.437974 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.437976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.438427 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.438433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.439225 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.439227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.439334 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.439758 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.439769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.442499 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.444159 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.444162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.444164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.444398 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.444689 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.444693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.444695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.444779 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.445183 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.445185 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.445255 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.445353 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.445466 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":32}
I20260619 14:59:35.445703 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.435714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.10.71.189:40263)
0619 14:59:35.435999 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.436011 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.436034 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:59:35.436077 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.436084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.436276 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.436435 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.436718 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.437857 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.438541 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.439108 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.439263 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.442619 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.442630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.442640 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.442661 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.443991 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.444503 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.444520 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.444521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.444596 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.444876 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.444880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.444882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.445079 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.445208 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.445416 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.445553 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab.
Entry for principal kudu/127.10.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab.
I20260619 14:59:35.447027 24004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.447306 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:46541
--webserver_interface=127.10.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab
--principal=kudu/127.10.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:34389,127.10.71.189:40263,127.10.71.188:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:35.447777 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.447339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48390)
0619 14:59:35.447621 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.447694 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
W20260619 14:59:35.447942 23953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
I20260619 14:59:35.448647 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.448467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48394)
0619 14:59:35.448562 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.448599 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:59:35.448776 24004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:46541: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
I20260619 14:59:35.461762 23929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } attempt: 3
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 14:59:35.468583 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.462104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54604)
0619 14:59:35.462200 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.462213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.462283 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.462516 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.462518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.462649 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.462729 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.462733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.463335 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.463338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.463836 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.463843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.464314 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.464318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.464422 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.464852 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.464861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.465931 (+ 1070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.467406 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.467408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.467409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.467572 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.467831 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.467833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.467834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.467903 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.468216 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.468218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.468282 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.468410 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.468492 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 14:59:35.468672 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.462154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.10.71.189:40263)
0619 14:59:35.462252 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.462257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.462267 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:59:35.462386 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.462390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.462429 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.462495 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.462825 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.463242 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.463950 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.464193 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.464297 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.466043 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.466047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.466049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.466069 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.467312 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.467651 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.467654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.467655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.467725 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.467974 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.467977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.467978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.468139 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.468221 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.468461 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.468562 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:59:35.469597 23929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.470124 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.469932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48404)
0619 14:59:35.470025 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.470071 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 14:59:35.470863 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.470664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48418)
0619 14:59:35.470777 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.470818 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 14:59:35.470991 23929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:46541: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
I20260619 14:59:35.497964 24004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } attempt: 1
I20260619 14:59:35.498478 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.498283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48428)
0619 14:59:35.498382 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.498429 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:59:35.499135 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.498849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48438)
0619 14:59:35.499029 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.499090 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 14:59:35.499243 24004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:46541: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
I20260619 14:59:35.502139 23929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } attempt: 1
I20260619 14:59:35.502682 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.502478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48452)
0619 14:59:35.502595 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.502635 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:59:35.503228 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.503055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48458)
0619 14:59:35.503145 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.503188 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:59:35.503334 23929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:46541: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
W20260619 14:59:35.545253 24011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:35.545472 24011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:35.545542 24011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:35.548980 24011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:35.549046 24011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:35.549113 24011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:35.549177 24011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:35.549211 24011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:35.553709 24011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:34389,127.10.71.189:40263,127.10.71.188:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab
--principal=kudu/127.10.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:35.554570 24011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:35.555418 24011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:35.562794 24016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:35.562834 24019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:35.562794 24011 server_base.cc:1034] running on GCE node
W20260619 14:59:35.562793 24017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:35.563373 24011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:35.564556 24011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:35.565728 24011 hybrid_clock.cc:648] HybridClock initialized: now 1781881175565720 us; error 41 us; skew 500 ppm
I20260619 14:59:35.567394 24004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } attempt: 2
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:35.568190 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.567805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48464)
0619 14:59:35.567982 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.568069 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 14:59:35.568259 24011 init.cc:377] Logged in from keytab as kudu/127.10.71.188@KRBTEST.COM (short username kudu)
I20260619 14:59:35.568910 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.568682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48474)
0619 14:59:35.568796 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.568848 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:59:35.569058 24004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:46541: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
I20260619 14:59:35.569283 24011 webserver.cc:460] Webserver started at http://127.10.71.188:35325/ using document root <none> and password file <none>
I20260619 14:59:35.569778 24011 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:35.569855 24011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:35.570079 24011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:35.571933 24011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data/instance:
uuid: "7739c1093f4340cc88df4b2441213aab"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "4b7dacdddc4676694a3090c27cfe5883"
server_key_iv: "bd7e56390bbf6235c8bd31cade4f0daa"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.572453 24011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal/instance:
uuid: "7739c1093f4340cc88df4b2441213aab"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "4b7dacdddc4676694a3090c27cfe5883"
server_key_iv: "bd7e56390bbf6235c8bd31cade4f0daa"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.575887 24011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:35.578073 24026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:35.578471 23929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } attempt: 2
I20260619 14:59:35.579103 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.578851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48476)
0619 14:59:35.578979 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.579035 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:59:35.579303 24011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:35.579398 24011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal
uuid: "7739c1093f4340cc88df4b2441213aab"
format_stamp: "Formatted at 2026-06-19 14:59:35 on dist-test-slave-3qdh"
server_key: "4b7dacdddc4676694a3090c27cfe5883"
server_key_iv: "bd7e56390bbf6235c8bd31cade4f0daa"
server_key_version: "encryptionkey@0"
I20260619 14:59:35.579493 24011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:35.579738 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.579555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48484)
0619 14:59:35.579638 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.579681 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 14:59:35.579881 23929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:46541: Network error: Client connection negotiation failed: client connection to 127.10.71.188:46541: connect: Connection refused (error 111)
I20260619 14:59:35.592849 24011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:35.595294 24011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:35.595453 24011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:35.601490 24011 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.188:46541
I20260619 14:59:35.601507 24078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.188:46541 every 8 connection(s)
I20260619 14:59:35.602247 24011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
I20260619 14:59:35.604575 24079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:35.607766 24079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.612120 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24011
I20260619 14:59:35.612264 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781881028756249-10526-0/minicluster-data/master-2/wal/instance
I20260619 14:59:35.612654 10526 external_mini_cluster.cc:1428] Setting key 615786f7f66c5c43601abae856d472a9
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 14:59:35.619475 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.609295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.10.71.190:34389)
0619 14:59:35.609397 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.609400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.610077 (+ 677us) server_negotiation.cc:407] Connection header received
0619 14:59:35.610795 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.610797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.610836 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.610897 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.611803 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.612345 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.613785 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.614012 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.614121 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.616184 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.616188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.616190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.616212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.617513 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.617953 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.617957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.617959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.618036 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.618402 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.618407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.618409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.618630 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.618727 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.619218 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.619328 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:59:35.620199 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.609608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34389 (local address 127.0.0.1:54624)
0619 14:59:35.609950 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.609985 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.610629 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.611019 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.611024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.611329 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.611697 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.611707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.612486 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.612490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.613661 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.613670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.614149 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.614154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.614323 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.614859 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.614872 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.616057 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.617619 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.617631 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.617642 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.617861 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.618124 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.618127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.618129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.618309 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.618737 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.618745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.618949 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.619135 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.619403 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"spinlock_wait_cycles":896,"thread_start_us":88,"threads_started":1}
I20260619 14:59:35.621439 24079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.622648 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.613252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.10.71.188:46541)
0619 14:59:35.613642 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.613655 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.613682 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:59:35.613726 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.613733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.613876 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.614029 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.614716 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.615365 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.616171 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.616618 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.616714 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.618934 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.618942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.618950 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.618971 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.620891 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.621272 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.621278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.621281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.621371 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.621633 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.621635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.621637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.621916 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.622093 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.622307 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.622438 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:35.629534 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.621846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:40263 (local address 127.0.0.1:54608)
0619 14:59:35.622080 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.622094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.622172 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.622407 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.622410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.622550 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.622653 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.622658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.623320 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.623325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.623878 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.623886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.624477 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.624481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.624569 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.624995 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.625006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.626174 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.628107 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.628109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.628111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.628363 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.628686 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.628688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.628689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.628758 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.629185 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.629189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.629277 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.629367 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.629446 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260619 14:59:35.629830 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.621950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.10.71.189:40263)
0619 14:59:35.622081 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.622085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.622137 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:59:35.622250 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.622253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.622297 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.622375 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.622760 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.623219 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.623991 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.624243 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.624366 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.626303 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.626306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.626308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.626332 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.627980 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.628469 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.628473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.628475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.628585 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.628849 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.628855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.628857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.629066 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.629167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.629451 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.629566 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:59:35.630407 24079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:59:35.632656 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.624394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.10.71.190:34389)
0619 14:59:35.624521 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.624525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.624537 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:59:35.624617 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.624622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.624657 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.624746 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.625275 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.625802 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.626746 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.626986 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.627121 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.629662 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.629666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.629672 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.629699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.631142 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.631568 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.631572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.631573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.631651 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.631905 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.631910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.631911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.632085 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.632287 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.632428 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.632526 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 14:59:35.637517 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.630846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.10.71.188:46541)
0619 14:59:35.630926 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.630929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.631035 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:59:35.631125 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.631128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.631160 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.631216 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.631525 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.631998 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.632734 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.633042 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.633154 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.634824 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.634827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.634830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.634864 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.636009 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.636435 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.636440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.636444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.636522 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.636808 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.636812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.636815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.637000 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.637076 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.637352 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.637436 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:59:35.637706 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.630811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48508)
0619 14:59:35.630975 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.630983 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.631065 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.631221 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.631223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.631361 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.631428 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.631432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.632091 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.632096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.632623 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.632632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.633169 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.633173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.633249 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.633656 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.633671 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.634709 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.636107 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.636110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.636111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.636298 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.636624 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.636627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.636628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.636719 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.637098 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.637102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.637205 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.637325 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.637530 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":68}
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 14:59:35.641144 24079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: Bootstrap starting.
I20260619 14:59:35.641851 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.633620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.10.71.189:40263)
0619 14:59:35.633764 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.633769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.633812 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:59:35.633919 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.633923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.633957 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.634076 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.634658 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.635352 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.636090 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.636407 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.636501 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.638857 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.638860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.638862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.638882 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.640371 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.640710 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.640713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.640714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.640788 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.641011 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.641013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.641015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.641210 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.641398 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.641689 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.641764 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:59:35.643728 24079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:35.644660 24079 log.cc:824] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:35.645313 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:59:35.646466 24079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: No bootstrap required, opened a new log
I20260619 14:59:35.650369 24079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.650593 24079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:35.650648 24079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7739c1093f4340cc88df4b2441213aab, State: Initialized, Role: FOLLOWER
I20260619 14:59:35.651078 24079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.652361 24079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:35.654688 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.656100 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.656730 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.647490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.10.71.189:40263)
0619 14:59:35.647578 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.647581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.648055 (+ 474us) server_negotiation.cc:407] Connection header received
0619 14:59:35.648238 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.648243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.648308 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.648392 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.649257 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.649975 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.651012 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.651334 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.651408 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.653128 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.653131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.653134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.653159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.654750 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.655454 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.655460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.655462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.655536 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.655822 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.655825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.655826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.655997 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.656095 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.656507 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.656605 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:59:35.657403 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.647099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.10.71.190:34389)
0619 14:59:35.647204 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.647209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.647415 (+ 206us) server_negotiation.cc:407] Connection header received
0619 14:59:35.647570 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.647573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.647612 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.647657 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.648621 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.649350 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.650445 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.650734 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.650793 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.652976 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.652979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.652981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.653003 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.655431 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.656234 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.656236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.656238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.656329 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.656592 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.656595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.656596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.656754 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.656838 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.657160 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.657280 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:59:35.657651 24103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:59:35.657704 24103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:59:35.656651 24011 master_runner.cc:186] Error getting master registration for 127.10.71.190:34389: OK, instance_id { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" instance_seqno: 1781881175222062 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:59:35.659951 24011 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:59:35.660204 24004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } attempt: 3
I20260619 14:59:35.661199 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.647223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.10.71.188:46541)
0619 14:59:35.647543 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.647547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.648019 (+ 472us) server_negotiation.cc:407] Connection header received
0619 14:59:35.648190 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.648193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.648228 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.648294 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.649159 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.649879 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.651129 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.651411 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.651474 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.653330 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.653337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.653343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.653369 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.655037 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.655512 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.655515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.655516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.655595 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.660043 (+ 4448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.660046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.660048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.660276 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.660441 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.660980 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.661077 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 14:59:35.667974 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.660558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48532)
0619 14:59:35.660704 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.660717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.660783 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.661012 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.661015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.661165 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.661222 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.661226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.662007 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.662010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.662483 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.662490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.662963 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.662966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.663046 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.663476 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.663489 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.664580 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.666688 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.666692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.666694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.666863 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.667109 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.667113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.667115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.667196 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.667515 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.667518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.667595 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.667691 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.667830 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 14:59:35.668045 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.660602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.10.71.188:46541)
0619 14:59:35.660832 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.660843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.660855 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:59:35.660880 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.660882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.660919 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.661001 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.661361 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.661893 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.662599 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.662857 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.662961 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.664717 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.664720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.664723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.664751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.666590 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.666951 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.666955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.666956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.667028 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.667282 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.667285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.667287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.667429 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.667520 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.667770 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.667881 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260619 14:59:35.670549 24004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: Bootstrap starting.
I20260619 14:59:35.672778 24004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:35.673496 24004 log.cc:824] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:35.675414 24004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: No bootstrap required, opened a new log
I20260619 14:59:35.678933 24004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.679044 23929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } attempt: 3
I20260619 14:59:35.679124 24004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:35.679164 24004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a638e79c7bb426da569b74148bbb42d, State: Initialized, Role: FOLLOWER
I20260619 14:59:35.679581 24004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.680076 24107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.680233 24107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.680640 24004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 14:59:35 dist-test-slave-3qdh krb5kdc[23848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881175, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 14:59:35.683264 23936 master_runner.cc:186] Error getting master registration for 127.10.71.190:34389: OK, instance_id { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" instance_seqno: 1781881175222062 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:59:35.686663 23936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:59:35.687266 24118 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:59:35.687326 24118 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:59:35.687470 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.679474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:46541 (local address 127.0.0.1:48542)
0619 14:59:35.679664 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:35.679677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:35.679736 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:35.680208 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:35.680211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:35.680386 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:35.680458 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.680463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.681106 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.681111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.681729 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:35.681738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.682249 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.682252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:35.682328 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.682809 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:35.682822 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:35.684220 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:35.686068 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.686072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.686074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.686304 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.686567 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:35.686570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:35.686571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:35.686646 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:35.687039 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:35.687044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:35.687126 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 14:59:35.687220 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:59:35.687321 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 14:59:35.687729 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:35.679861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.10.71.188:46541)
0619 14:59:35.679951 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:35.679955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:35.679967 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:59:35.680007 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:35.680009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:35.680041 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:35.680103 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:35.680559 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.681015 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.681829 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:35.682150 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:35.682240 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:35.684375 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:35.684379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:35.684381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:35.684407 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:35.685958 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.686398 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.686401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.686402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.686480 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:35.686770 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:35.686773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:35.686774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:35.686941 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:35.687024 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:35.687481 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 14:59:35.687590 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:59:35.690184 23929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: Bootstrap starting.
I20260619 14:59:35.692312 23929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:35.692998 23929 log.cc:824] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:35.697741 23929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: No bootstrap required, opened a new log
I20260619 14:59:35.700991 23929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.701159 23929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:35.701195 23929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574601dab7ea4ad8bdb1613510da7c06, State: Initialized, Role: FOLLOWER
I20260619 14:59:35.701539 23929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.702175 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.702935 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.703639 23929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:59:35.707845 24132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:59:35.707953 24132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:59:35.710525 23861 master_runner.cc:263] No leader master found from master 574601dab7ea4ad8bdb1613510da7c06
I20260619 14:59:35.710582 23861 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:59:35.719003 23882 tablet.cc:2369] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:35.767753 24121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:35.767899 24121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.768944 24121 leader_election.cc:288] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a638e79c7bb426da569b74148bbb42d (127.10.71.189:40263), 7739c1093f4340cc88df4b2441213aab (127.10.71.188:46541)
I20260619 14:59:35.769477 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "574601dab7ea4ad8bdb1613510da7c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7739c1093f4340cc88df4b2441213aab" is_pre_election: true
I20260619 14:59:35.769771 24053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 574601dab7ea4ad8bdb1613510da7c06 in term 0.
I20260619 14:59:35.770138 23878 leader_election.cc:302] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574601dab7ea4ad8bdb1613510da7c06, 7739c1093f4340cc88df4b2441213aab; no voters:
I20260619 14:59:35.770722 23978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "574601dab7ea4ad8bdb1613510da7c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a638e79c7bb426da569b74148bbb42d" is_pre_election: true
I20260619 14:59:35.770936 23978 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 574601dab7ea4ad8bdb1613510da7c06 in term 0.
I20260619 14:59:35.771379 24121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:59:35.771453 24121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:59:35.771507 24121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:35.772706 24121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.772998 24121 leader_election.cc:288] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [CANDIDATE]: Term 1 election: Requested vote from peers 4a638e79c7bb426da569b74148bbb42d (127.10.71.189:40263), 7739c1093f4340cc88df4b2441213aab (127.10.71.188:46541)
I20260619 14:59:35.773485 23978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "574601dab7ea4ad8bdb1613510da7c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a638e79c7bb426da569b74148bbb42d"
I20260619 14:59:35.773586 23978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:35.774788 23978 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 574601dab7ea4ad8bdb1613510da7c06 in term 1.
I20260619 14:59:35.775105 23880 leader_election.cc:302] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a638e79c7bb426da569b74148bbb42d, 574601dab7ea4ad8bdb1613510da7c06; no voters:
I20260619 14:59:35.775290 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "574601dab7ea4ad8bdb1613510da7c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7739c1093f4340cc88df4b2441213aab"
I20260619 14:59:35.775393 24053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:35.778059 24053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 574601dab7ea4ad8bdb1613510da7c06 in term 1.
I20260619 14:59:35.778919 24121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:35.779152 24121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [term 1 LEADER]: Becoming Leader. State: Replica: 574601dab7ea4ad8bdb1613510da7c06, State: Running, Role: LEADER
I20260619 14:59:35.779448 24121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } }
I20260619 14:59:35.781075 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574601dab7ea4ad8bdb1613510da7c06. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.781175 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:35.781605 24136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:35.782678 24136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:35.789011 24053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [term 1 FOLLOWER]: Refusing update from remote peer 574601dab7ea4ad8bdb1613510da7c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:35.789597 24121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:35.793301 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574601dab7ea4ad8bdb1613510da7c06. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.793411 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.794389 23978 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [term 1 FOLLOWER]: Refusing update from remote peer 574601dab7ea4ad8bdb1613510da7c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:35.796883 24121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:35.799734 24107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574601dab7ea4ad8bdb1613510da7c06. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.799834 24107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.803489 24134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.803587 24134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:35.804723 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.804829 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.805354 24137 mvcc.cc:205] Tried to move back new op lower bound from 7298585296024752128 to 7298585295995121664. Current Snapshot: MvccSnapshot[applied={T|T < 7298585296024752128}]
I20260619 14:59:35.805689 24138 mvcc.cc:205] Tried to move back new op lower bound from 7298585296024752128 to 7298585295995121664. Current Snapshot: MvccSnapshot[applied={T|T < 7298585296024752128}]
I20260619 14:59:35.807061 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.807159 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:35.808074 24107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "574601dab7ea4ad8bdb1613510da7c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574601dab7ea4ad8bdb1613510da7c06" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34389 } } peers { permanent_uuid: "4a638e79c7bb426da569b74148bbb42d" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 40263 } } peers { permanent_uuid: "7739c1093f4340cc88df4b2441213aab" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 46541 } } }
I20260619 14:59:35.808178 24107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:59:35.810367 24136 catalog_manager.cc:1347] Generated new cluster ID: b4ce6fa53edc457ba2a0516ce80642cf
I20260619 14:59:35.810424 24136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:35.812357 24140 mvcc.cc:205] Tried to move back new op lower bound from 7298585296024752128 to 7298585295995121664. Current Snapshot: MvccSnapshot[applied={T|T < 7298585296024752128}]
I20260619 14:59:35.831257 24136 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:59:35.832788 24136 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:35.843181 24136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 574601dab7ea4ad8bdb1613510da7c06: Generated new TSK 0
I20260619 14:59:35.843819 24136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:59:36.658802 24103 catalog_manager.cc:1259] Loaded cluster ID: b4ce6fa53edc457ba2a0516ce80642cf
I20260619 14:59:36.658921 24103 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: loading cluster ID for follower catalog manager: success
I20260619 14:59:36.660581 24103 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: acquiring CA information for follower catalog manager: success
I20260619 14:59:36.661691 24103 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7739c1093f4340cc88df4b2441213aab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:59:36.688277 24118 catalog_manager.cc:1259] Loaded cluster ID: b4ce6fa53edc457ba2a0516ce80642cf
I20260619 14:59:36.688347 24118 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: loading cluster ID for follower catalog manager: success
I20260619 14:59:36.689740 24118 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: acquiring CA information for follower catalog manager: success
I20260619 14:59:36.690757 24118 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a638e79c7bb426da569b74148bbb42d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:59:36.902065 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23861
I20260619 14:59:36.907555 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 23936
I20260619 14:59:36.912940 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24011
2026-06-19T14:59:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2921 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:59:36 dist-test-slave-3qdh krb5kdc[24151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:59:36 dist-test-slave-3qdh krb5kdc[24151](info): set up 2 sockets
Jun 19 14:59:36 dist-test-slave-3qdh krb5kdc[24151](info): commencing operation
krb5kdc: starting...
W20260619 14:59:37.876601 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:59:37 dist-test-slave-3qdh krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:59:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:59:37.991670 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40155
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:38.093591 24164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:38.093833 24164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:38.093925 24164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:38.097678 24164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:38.097752 24164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:38.097788 24164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:38.097841 24164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:38.097894 24164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:38.102383 24164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:38.103262 24164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:38.104195 24164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:38.111532 24169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:38.111618 24164 server_base.cc:1034] running on GCE node
W20260619 14:59:38.111624 24170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:38.111542 24172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:38.112251 24164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:38.113226 24164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:38.114400 24164 hybrid_clock.cc:648] HybridClock initialized: now 1781881178114383 us; error 27 us; skew 500 ppm
Jun 19 14:59:38 dist-test-slave-3qdh krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881178, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:38.116834 24164 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:38.117547 24164 webserver.cc:460] Webserver started at http://127.10.71.190:40001/ using document root <none> and password file <none>
I20260619 14:59:38.117998 24164 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:38.118084 24164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:38.118278 24164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:38.119938 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "d10d4aef51414879b44804f769ad2228"
format_stamp: "Formatted at 2026-06-19 14:59:38 on dist-test-slave-3qdh"
server_key: "395ac0744638476217325da622e6950d"
server_key_iv: "dd6a7983772597ac681b918cb01d0cfc"
server_key_version: "encryptionkey@0"
I20260619 14:59:38.120471 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "d10d4aef51414879b44804f769ad2228"
format_stamp: "Formatted at 2026-06-19 14:59:38 on dist-test-slave-3qdh"
server_key: "395ac0744638476217325da622e6950d"
server_key_iv: "dd6a7983772597ac681b918cb01d0cfc"
server_key_version: "encryptionkey@0"
I20260619 14:59:38.123479 24164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:59:38.125662 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:38.126632 24164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:59:38.126757 24164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "d10d4aef51414879b44804f769ad2228"
format_stamp: "Formatted at 2026-06-19 14:59:38 on dist-test-slave-3qdh"
server_key: "395ac0744638476217325da622e6950d"
server_key_iv: "dd6a7983772597ac681b918cb01d0cfc"
server_key_version: "encryptionkey@0"
I20260619 14:59:38.126854 24164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:38.150154 24164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:38.152753 24164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:38.152935 24164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:38.159802 24164 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:40155
I20260619 14:59:38.159830 24231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:40155 every 8 connection(s)
I20260619 14:59:38.160636 24164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:38.163096 24232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:38.166977 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24164
I20260619 14:59:38.167104 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:59:38.167415 10526 external_mini_cluster.cc:1428] Setting key 1370ea5e6c126d483d18778c08ccbf27
I20260619 14:59:38.168776 24232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228: Bootstrap starting.
I20260619 14:59:38.170842 24232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:38.171538 24232 log.cc:824] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:38.173310 24232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228: No bootstrap required, opened a new log
Jun 19 14:59:38 dist-test-slave-3qdh krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:38.177022 24232 raft_consensus.cc:348] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10d4aef51414879b44804f769ad2228" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40155 } }
I20260619 14:59:38.177206 24232 raft_consensus.cc:374] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:38.177239 24232 raft_consensus.cc:729] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10d4aef51414879b44804f769ad2228, State: Initialized, Role: FOLLOWER
I20260619 14:59:38.177685 24232 consensus_queue.cc:260] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10d4aef51414879b44804f769ad2228" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40155 } }
I20260619 14:59:38.177800 24232 raft_consensus.cc:388] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:38.177881 24232 raft_consensus.cc:482] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:38.177986 24232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:38.178802 24232 raft_consensus.cc:504] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10d4aef51414879b44804f769ad2228" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40155 } }
I20260619 14:59:38.179136 24232 leader_election.cc:302] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d10d4aef51414879b44804f769ad2228; no voters:
I20260619 14:59:38.179415 24232 leader_election.cc:288] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:59:38.179543 24237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:38.179833 24237 raft_consensus.cc:686] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [term 1 LEADER]: Becoming Leader. State: Replica: d10d4aef51414879b44804f769ad2228, State: Running, Role: LEADER
I20260619 14:59:38.180194 24237 consensus_queue.cc:237] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10d4aef51414879b44804f769ad2228" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40155 } }
I20260619 14:59:38.180555 24232 sys_catalog.cc:564] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:38.181406 24237 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d10d4aef51414879b44804f769ad2228. Latest consensus state: current_term: 1 leader_uuid: "d10d4aef51414879b44804f769ad2228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10d4aef51414879b44804f769ad2228" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40155 } } }
I20260619 14:59:38.182181 24237 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:38.182070 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d10d4aef51414879b44804f769ad2228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10d4aef51414879b44804f769ad2228" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40155 } } }
I20260619 14:59:38.182493 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:38.182502 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:38.168749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.10.71.190:40155)
0619 14:59:38.169133 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:38.169146 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:38.169179 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:59:38.169750 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:38.169764 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:38.170041 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:38.170336 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:38.171128 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:38.172074 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:38.172958 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:38.173665 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:38.173837 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:38.176543 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:38.176548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:38.176558 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:38.176581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:38.178472 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:38.178984 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:38.178990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:38.178995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:38.179164 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:38.179686 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:38.179690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:38.179692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:38.180232 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:38.181033 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:38.181201 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 14:59:38.181475 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
W20260619 14:59:38.185505 24252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:59:38.185595 24252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:59:38.185702 24251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:38.186230 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:59:38.186678 24251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:38.191653 24251 catalog_manager.cc:1347] Generated new cluster ID: 6db749c0af3b44d88be3ff0a8f41982e
I20260619 14:59:38.191713 24251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:38.203100 24251 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:59:38.204157 24251 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:38.211601 24251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d10d4aef51414879b44804f769ad2228: Generated new TSK 0
I20260619 14:59:38.212253 24251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:59:38.241904 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24164
2026-06-19T14:59:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1328 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:59:38 dist-test-slave-3qdh krb5kdc[24259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:59:38 dist-test-slave-3qdh krb5kdc[24259](info): set up 2 sockets
Jun 19 14:59:38 dist-test-slave-3qdh krb5kdc[24259](info): commencing operation
krb5kdc: starting...
W20260619 14:59:39.264775 10526 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:59:39.370882 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42951
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:39.464650 24272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:39.464898 24272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:39.464969 24272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:39.468561 24272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:39.468636 24272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:39.468672 24272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:39.468724 24272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:39.468761 24272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:39.473467 24272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:42951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:39.474298 24272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:39.475163 24272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:39.482584 24280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:39.482611 24278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:39.482615 24272 server_base.cc:1034] running on GCE node
W20260619 14:59:39.482630 24277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:39.483150 24272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:39.484037 24272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:39.485198 24272 hybrid_clock.cc:648] HybridClock initialized: now 1781881179485173 us; error 41 us; skew 500 ppm
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:39.487758 24272 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:39.488595 24272 webserver.cc:460] Webserver started at http://127.10.71.190:46529/ using document root <none> and password file <none>
I20260619 14:59:39.489065 24272 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:39.489146 24272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:39.489344 24272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:39.491053 24272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "63d8ecf3d62147618934619f1449ef60"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "5e6e5616a118cca594e66d382f47e984"
server_key_iv: "6c07dd314275874ec499df0cfd1a9f03"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.491569 24272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "63d8ecf3d62147618934619f1449ef60"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "5e6e5616a118cca594e66d382f47e984"
server_key_iv: "6c07dd314275874ec499df0cfd1a9f03"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.494740 24272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:59:39.496801 24287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.497859 24272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:39.497990 24272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "63d8ecf3d62147618934619f1449ef60"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "5e6e5616a118cca594e66d382f47e984"
server_key_iv: "6c07dd314275874ec499df0cfd1a9f03"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.498087 24272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:39.515425 24272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:39.517607 24272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:39.517774 24272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:39.523854 24272 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:42951
I20260619 14:59:39.524132 24339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:42951 every 8 connection(s)
I20260619 14:59:39.524833 24272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:39.527113 24340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:39.531869 24340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap starting.
I20260619 14:59:39.533887 24340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:39.534583 24340 log.cc:824] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:39.535128 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24272
I20260619 14:59:39.535279 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:59:39.535526 10526 external_mini_cluster.cc:1428] Setting key 74447c3c8b32e68fbecc4712056dc3ae
I20260619 14:59:39.536535 24340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: No bootstrap required, opened a new log
I20260619 14:59:39.539918 24340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:39.540087 24340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:39.540117 24340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d8ecf3d62147618934619f1449ef60, State: Initialized, Role: FOLLOWER
I20260619 14:59:39.540589 24340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:39.540730 24340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:39.540788 24340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:39.540874 24340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:39.541736 24340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:39.542063 24340 leader_election.cc:302] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d8ecf3d62147618934619f1449ef60; no voters:
I20260619 14:59:39.542357 24340 leader_election.cc:288] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:59:39.542639 24346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:39.542834 24346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 LEADER]: Becoming Leader. State: Replica: 63d8ecf3d62147618934619f1449ef60, State: Running, Role: LEADER
I20260619 14:59:39.543107 24346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:39.543378 24340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:39.544113 24346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d8ecf3d62147618934619f1449ef60. Latest consensus state: current_term: 1 leader_uuid: "63d8ecf3d62147618934619f1449ef60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } } }
I20260619 14:59:39.544227 24346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:39.544770 24345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d8ecf3d62147618934619f1449ef60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } } }
I20260619 14:59:39.544867 24345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:39.546008 24354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:39.549130 24354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:39.549587 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:39.536531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.10.71.190:42951)
0619 14:59:39.536963 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:39.536978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:39.537023 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:59:39.537658 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:39.537673 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:39.537944 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:39.538224 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:39.539081 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.540360 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.541459 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.541922 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.542053 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:39.544931 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:39.544937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:39.544951 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:39.544981 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:39.547074 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:39.547459 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:39.547464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:39.547471 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:39.547574 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:39.547833 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:39.547837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:39.547838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:39.548139 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:39.548380 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:39.548595 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:59:39.548935 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":99,"threads_started":1}
I20260619 14:59:39.553840 24354 catalog_manager.cc:1347] Generated new cluster ID: 88307da44c244989a384ce0affa49aac
I20260619 14:59:39.553891 24354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:39.572151 24354 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:59:39.572870 24354 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:39.581285 24354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Generated new TSK 0
I20260619 14:59:39.582091 24354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:59:39.607976 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42951
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:39.704531 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:39.704749 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:39.704813 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:39.708467 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:39.708518 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:39.708601 24366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:59:39.713143 24366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:39.714076 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:39.715320 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:39.724232 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:39.724243 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:39.724255 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:39.724433 24366 server_base.cc:1034] running on GCE node
I20260619 14:59:39.724846 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:39.725435 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:39.726611 24366 hybrid_clock.cc:648] HybridClock initialized: now 1781881179726599 us; error 33 us; skew 500 ppm
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:39.729395 24366 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:59:39.730134 24366 webserver.cc:460] Webserver started at http://127.10.71.129:36335/ using document root <none> and password file <none>
I20260619 14:59:39.730624 24366 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:39.730706 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:39.730909 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:39.732728 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "3dbef50bf0444769b9bd4e2eda487dfc"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "463917dec6a7a021a22042249b9cb24a"
server_key_iv: "0adcdad873d9358a7f96fe3cea96aad9"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.733268 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "3dbef50bf0444769b9bd4e2eda487dfc"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "463917dec6a7a021a22042249b9cb24a"
server_key_iv: "0adcdad873d9358a7f96fe3cea96aad9"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.736511 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:39.738549 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.739705 24366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:39.739835 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "3dbef50bf0444769b9bd4e2eda487dfc"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "463917dec6a7a021a22042249b9cb24a"
server_key_iv: "0adcdad873d9358a7f96fe3cea96aad9"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.739931 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:39.762151 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:39.765061 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:39.765264 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:39.765735 24366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:39.766448 24366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:39.766516 24366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.766587 24366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:39.766630 24366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.774587 24366 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:45727
I20260619 14:59:39.774633 24494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:45727 every 8 connection(s)
I20260619 14:59:39.775511 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:59:39.782696 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24366
I20260619 14:59:39.782824 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:59:39.783114 10526 external_mini_cluster.cc:1428] Setting key 6c133df4ec8d8a0b880a680eb1b69860
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:39.787833 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:39.777106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:50467 (local address 127.10.71.190:42951)
0619 14:59:39.777223 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:39.777226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:39.777851 (+ 625us) server_negotiation.cc:407] Connection header received
0619 14:59:39.779090 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:39.779093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:39.779149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:39.779258 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:39.780236 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.780762 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.781791 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.782013 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.782117 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:39.784567 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:39.784571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:39.784573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:39.784604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:39.785836 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:39.786363 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:39.786368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:39.786371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:39.786460 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:39.786748 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:39.786752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:39.786754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:39.786991 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:39.787088 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:39.787599 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 14:59:39.787696 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:59:39.788884 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:39.777352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:50467)
0619 14:59:39.777720 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:39.777754 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:39.778891 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:39.779410 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:39.779415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:39.779699 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:39.780100 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:39.780112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.780878 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.780881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:39.781667 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:39.781676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.782146 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.782150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:39.782312 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:39.782803 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:39.782823 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:39.784412 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:39.786004 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:39.786012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:39.786028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:39.786269 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:39.786558 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:39.786560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:39.786561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:39.786668 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:39.787111 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:39.787119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:39.787377 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 14:59:39.787558 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 14:59:39.787888 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260619 14:59:39.789978 24495 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42951
I20260619 14:59:39.790189 24495 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:39.790596 24495 heartbeater.cc:511] Master 127.10.71.190:42951 requested a full tablet report, sending...
I20260619 14:59:39.791937 24303 ts_manager.cc:194] Registered new tserver with Master: 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727)
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 14:59:39.793016 24303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:50467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:59:39.808990 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42951
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:39.905189 24500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:39.905413 24500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:39.905478 24500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:39.908875 24500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:39.908951 24500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:39.909042 24500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:59:39.913357 24500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:39.914186 24500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:39.915028 24500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:39.923094 24505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:39.923120 24506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:39.923167 24500 server_base.cc:1034] running on GCE node
W20260619 14:59:39.923120 24508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:39.923658 24500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:39.924238 24500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:39.925386 24500 hybrid_clock.cc:648] HybridClock initialized: now 1781881179925372 us; error 31 us; skew 500 ppm
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:39.927791 24500 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:59:39.928663 24500 webserver.cc:460] Webserver started at http://127.10.71.130:39471/ using document root <none> and password file <none>
I20260619 14:59:39.929126 24500 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:39.929177 24500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:39.929347 24500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:39.930967 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "6ffa6bed5f934e7cb4870a07766cb254"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "dfd364a2d33340cf1d74cac01c1eb468"
server_key_iv: "e3c5ffa809b1253b18fb1ca0c7a0fcf1"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.931430 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "6ffa6bed5f934e7cb4870a07766cb254"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "dfd364a2d33340cf1d74cac01c1eb468"
server_key_iv: "e3c5ffa809b1253b18fb1ca0c7a0fcf1"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.934476 24500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:59:39.936642 24515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.937636 24500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:39.937767 24500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "6ffa6bed5f934e7cb4870a07766cb254"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "dfd364a2d33340cf1d74cac01c1eb468"
server_key_iv: "e3c5ffa809b1253b18fb1ca0c7a0fcf1"
server_key_version: "encryptionkey@0"
I20260619 14:59:39.937860 24500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:39.955685 24500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:39.958554 24500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:39.958739 24500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:39.959228 24500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:39.959915 24500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:39.959990 24500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.960049 24500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:39.960094 24500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:39.968770 24500 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:33199
I20260619 14:59:39.968789 24628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:33199 every 8 connection(s)
I20260619 14:59:39.969578 24500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:59:39.973841 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24500
I20260619 14:59:39.973975 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:59:39.974304 10526 external_mini_cluster.cc:1428] Setting key f5f94e88f9196ae5375ee0ea36349e42
Jun 19 14:59:39 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881179, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:39.983160 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:39.971142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:50559 (local address 127.10.71.190:42951)
0619 14:59:39.971282 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:39.971285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:39.971969 (+ 684us) server_negotiation.cc:407] Connection header received
0619 14:59:39.973313 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:39.973317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:39.973375 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:39.973494 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:39.974918 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.975576 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.976655 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.977030 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.977167 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:39.979826 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:39.979829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:39.979831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:39.979851 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:39.981139 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:39.981670 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:39.981673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:39.981674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:39.981753 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:39.982101 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:39.982103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:39.982104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:39.982269 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:39.982335 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:39.982949 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 14:59:39.983044 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 14:59:39.984104 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:39.971414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:50559)
0619 14:59:39.971824 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:39.971861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:39.973112 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:39.973681 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:39.973691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:39.974181 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:39.974775 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:39.974790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.975711 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.975713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:39.976554 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:39.976562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:39.977217 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:39.977222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:39.977415 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:39.977931 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:39.977951 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:39.979678 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:39.981241 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:39.981248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:39.981262 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:39.981576 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:39.981839 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:39.981843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:39.981846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:39.982005 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:39.982367 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:39.982373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:39.982655 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 14:59:39.982897 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 14:59:39.983230 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
I20260619 14:59:39.985149 24629 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42951
I20260619 14:59:39.985396 24629 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 14:59:39.985824 24629 heartbeater.cc:511] Master 127.10.71.190:42951 requested a full tablet report, sending...
I20260619 14:59:39.986807 24303 ts_manager.cc:194] Registered new tserver with Master: 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199)
I20260619 14:59:39.987517 24303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:50559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:59:39.999214 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42951
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:40.098317 24634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:40.098546 24634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:40.098606 24634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:40.102495 24634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:40.102561 24634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:40.102644 24634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:59:40.107225 24634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:40.108111 24634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:40.109032 24634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:40.117524 24642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:40.117623 24634 server_base.cc:1034] running on GCE node
W20260619 14:59:40.117524 24639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.117537 24640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:40.118170 24634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:40.118717 24634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:40.119920 24634 hybrid_clock.cc:648] HybridClock initialized: now 1781881180119898 us; error 33 us; skew 500 ppm
Jun 19 14:59:40 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:40.122344 24634 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:59:40.123093 24634 webserver.cc:460] Webserver started at http://127.10.71.131:33865/ using document root <none> and password file <none>
I20260619 14:59:40.123584 24634 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:40.123683 24634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:40.123883 24634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:40.125792 24634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "8a00b83581344e6dbbe065a6ffb579db"
format_stamp: "Formatted at 2026-06-19 14:59:40 on dist-test-slave-3qdh"
server_key: "5c192e9e3da4ebca9ad082e20f212c7b"
server_key_iv: "916f76d234d99b4e0556f4d081354137"
server_key_version: "encryptionkey@0"
I20260619 14:59:40.126295 24634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "8a00b83581344e6dbbe065a6ffb579db"
format_stamp: "Formatted at 2026-06-19 14:59:40 on dist-test-slave-3qdh"
server_key: "5c192e9e3da4ebca9ad082e20f212c7b"
server_key_iv: "916f76d234d99b4e0556f4d081354137"
server_key_version: "encryptionkey@0"
I20260619 14:59:40.129662 24634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:59:40.131803 24649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:40.132862 24634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:59:40.133001 24634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "8a00b83581344e6dbbe065a6ffb579db"
format_stamp: "Formatted at 2026-06-19 14:59:40 on dist-test-slave-3qdh"
server_key: "5c192e9e3da4ebca9ad082e20f212c7b"
server_key_iv: "916f76d234d99b4e0556f4d081354137"
server_key_version: "encryptionkey@0"
I20260619 14:59:40.133106 24634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:40.154245 24634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:40.156759 24634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:40.156948 24634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:40.157693 24634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:40.158447 24634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:40.158517 24634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:40.158584 24634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:40.158633 24634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:40.167140 24634 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:36035
I20260619 14:59:40.167161 24762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:36035 every 8 connection(s)
I20260619 14:59:40.167901 24634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:59:40.174247 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24634
I20260619 14:59:40.174363 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:59:40.174643 10526 external_mini_cluster.cc:1428] Setting key 763304b4178ec1e0b0faa8c8250b0651
Jun 19 14:59:40 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:40.187211 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.169629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:59745 (local address 127.10.71.190:42951)
0619 14:59:40.169781 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.169786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.170479 (+ 693us) server_negotiation.cc:407] Connection header received
0619 14:59:40.171584 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.171587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.171629 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.171692 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:40.172708 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.173164 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.174199 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.174549 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.174664 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.184222 (+ 9558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:40.184225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:40.184227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:40.184249 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:40.185555 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:40.185950 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:40.185953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:40.185954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:40.186017 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:40.186271 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:40.186273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:40.186274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:40.186433 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:40.186498 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.186930 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.187053 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:59:40.188098 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.169923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:59745)
0619 14:59:40.170348 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.170384 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.171416 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.171817 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.171823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.172134 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:40.172591 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.172604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.173274 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.173276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.174043 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.174070 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.174742 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.174748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.174909 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.182667 (+ 7758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:40.182681 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:40.184062 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:40.185645 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:40.185649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:40.185658 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:40.185874 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:40.186087 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:40.186088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:40.186090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:40.186202 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:40.186517 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:40.186521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:40.186705 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.186895 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.187175 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260619 14:59:40.189046 24763 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42951
I20260619 14:59:40.189244 24763 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:40.189626 24763 heartbeater.cc:511] Master 127.10.71.190:42951 requested a full tablet report, sending...
I20260619 14:59:40.190439 24303 ts_manager.cc:194] Registered new tserver with Master: 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
I20260619 14:59:40.190935 24303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:59745
I20260619 14:59:40.199949 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:59:40.208223 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.201890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.10.71.190:42951)
0619 14:59:40.202020 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.202024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.202122 (+ 98us) server_negotiation.cc:407] Connection header received
0619 14:59:40.202209 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.202211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.202241 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.202295 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:40.202899 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.203352 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.204103 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.204348 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.204437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.205617 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:40.205620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:40.205621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:40.205640 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:40.206897 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:40.207218 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:40.207220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:40.207221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:40.207279 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:40.207463 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:40.207465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:40.207466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:40.207638 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:40.207722 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.207956 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.208063 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:59:40.210500 24303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:59:40.213192 24303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:59:40.226416 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.221125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.10.71.130:33199)
0619 14:59:40.221767 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.221777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.221795 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:59:40.221859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.221863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.222012 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.222135 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:40.222513 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.223157 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.224954 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.225708 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.225814 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.225938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.226212 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.226294 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":131,"threads_started":1}
I20260619 14:59:40.226431 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.220325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:45727 (local address 127.0.0.1:48126)
0619 14:59:40.220881 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.220934 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.221190 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.221873 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.221878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.221896 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:40.221989 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.221995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.223331 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.223334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.224711 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.224727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.225827 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.225830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.226009 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.226061 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.226169 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.226288 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":86,"thread_start_us":71,"threads_started":1}
I20260619 14:59:40.226432 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.220995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:33199 (local address 127.0.0.1:46812)
0619 14:59:40.221474 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.221485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.221564 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.222307 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.222309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.222320 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:40.222402 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.222410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.223283 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.223288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.224827 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.224838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.225827 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.225830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.225989 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.226062 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.226169 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.226288 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":227,"thread_start_us":83,"threads_started":1}
I20260619 14:59:40.226868 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.220448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.10.71.129:45727)
0619 14:59:40.220721 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.220733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.221316 (+ 583us) server_negotiation.cc:407] Connection header received
0619 14:59:40.221391 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.221396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.221582 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.221715 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:40.222112 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.223195 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.224855 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.225708 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.225814 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.225938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.226621 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.226727 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":117,"threads_started":1}
I20260619 14:59:40.227478 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.220433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36035 (local address 127.0.0.1:49954)
0619 14:59:40.221349 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.221363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.221431 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.222252 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.222255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.222269 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:40.222367 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.222372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.223513 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.223522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.224828 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.224838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.227093 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.227096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.227189 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.227215 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.227300 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.227362 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":77,"thread_start_us":78,"threads_started":1}
I20260619 14:59:40.227494 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.221042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.10.71.131:36035)
0619 14:59:40.221882 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.221889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.221904 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:59:40.221954 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.221957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.222106 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.222252 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:40.222471 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.223399 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.225998 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.226950 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.227065 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.227185 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.227306 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.227381 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":126,"threads_started":1}
I20260619 14:59:40.229653 24429 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 (DEFAULT_TABLE table=test-table [id=f036c68c9abd4a148ab90aefcd33f032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:40.229779 24697 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 (DEFAULT_TABLE table=test-table [id=f036c68c9abd4a148ab90aefcd33f032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:40.229779 24563 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 (DEFAULT_TABLE table=test-table [id=f036c68c9abd4a148ab90aefcd33f032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:40.230597 24429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf63a9d05194a9f8a6dde1837eb09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:40.230724 24697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf63a9d05194a9f8a6dde1837eb09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:40.230720 24563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf63a9d05194a9f8a6dde1837eb09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:40.236406 24780 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap starting.
I20260619 14:59:40.236999 24782 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Bootstrap starting.
I20260619 14:59:40.237489 24781 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Bootstrap starting.
I20260619 14:59:40.238642 24780 tablet_bootstrap.cc:654] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:40.238808 24782 tablet_bootstrap.cc:654] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:40.239472 24781 tablet_bootstrap.cc:654] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:40.239920 24782 log.cc:824] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:40.239920 24780 log.cc:824] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:40.240090 24781 log.cc:824] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:40.241672 24781 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: No bootstrap required, opened a new log
I20260619 14:59:40.241672 24780 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: No bootstrap required, opened a new log
I20260619 14:59:40.241693 24782 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: No bootstrap required, opened a new log
I20260619 14:59:40.241837 24782 ts_tablet_manager.cc:1397] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:59:40.241833 24781 ts_tablet_manager.cc:1397] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260619 14:59:40.241833 24780 ts_tablet_manager.cc:1397] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:59:40.244856 24782 raft_consensus.cc:348] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.245065 24782 raft_consensus.cc:374] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:40.245153 24782 raft_consensus.cc:729] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a00b83581344e6dbbe065a6ffb579db, State: Initialized, Role: FOLLOWER
I20260619 14:59:40.245409 24781 raft_consensus.cc:348] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.245404 24780 raft_consensus.cc:348] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.245617 24780 raft_consensus.cc:374] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:40.245617 24781 raft_consensus.cc:374] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:40.245574 24782 consensus_queue.cc:260] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.245667 24781 raft_consensus.cc:729] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ffa6bed5f934e7cb4870a07766cb254, State: Initialized, Role: FOLLOWER
I20260619 14:59:40.245714 24780 raft_consensus.cc:729] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbef50bf0444769b9bd4e2eda487dfc, State: Initialized, Role: FOLLOWER
I20260619 14:59:40.246084 24763 heartbeater.cc:503] Master 127.10.71.190:42951 was elected leader, sending a full tablet report...
I20260619 14:59:40.246069 24780 consensus_queue.cc:260] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.246069 24781 consensus_queue.cc:260] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.246393 24782 ts_tablet_manager.cc:1428] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:59:40.246879 24780 ts_tablet_manager.cc:1428] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:59:40.246879 24781 ts_tablet_manager.cc:1428] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:59:40.247192 24629 heartbeater.cc:503] Master 127.10.71.190:42951 was elected leader, sending a full tablet report...
I20260619 14:59:40.247192 24495 heartbeater.cc:503] Master 127.10.71.190:42951 was elected leader, sending a full tablet report...
W20260619 14:59:40.276532 24496 tablet.cc:2369] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:40.385633 24787 raft_consensus.cc:482] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:40.385901 24787 raft_consensus.cc:504] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.387171 24787 leader_election.cc:288] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
I20260619 14:59:40.390843 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.387527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:55737 (local address 127.10.71.131:36035)
0619 14:59:40.387658 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.387662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.387678 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:59:40.387796 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.387799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.387861 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.387948 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:40.388337 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.388886 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.389871 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.390269 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.390352 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.390416 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.390627 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.390721 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:59:40.390853 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.387475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:36023 (local address 127.10.71.129:45727)
0619 14:59:40.387612 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.387617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.388076 (+ 459us) server_negotiation.cc:407] Connection header received
0619 14:59:40.388139 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.388142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.388189 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.388275 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:40.388551 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.389017 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.389969 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.390331 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.390434 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.390480 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.390688 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.390758 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:59:40.390848 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.387420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36035 (local address 127.10.71.130:55737)
0619 14:59:40.387558 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.387574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.387671 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.387975 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.387977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.388019 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:40.388201 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.388210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.388985 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.388988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.389739 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.389748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.390390 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.390396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.390492 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.390518 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.390580 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.390684 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"spinlock_wait_cycles":14592}
I20260619 14:59:40.390861 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.387419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:45727 (local address 127.10.71.130:36023)
0619 14:59:40.387972 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.387985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.388078 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.388330 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.388333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.388347 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:40.388447 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.388452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.389126 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.389129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.389858 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.389867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.390458 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.390461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.390530 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.390545 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.390637 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.390720 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":121,"thread_start_us":140,"threads_started":1}
I20260619 14:59:40.391464 24449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf63a9d05194a9f8a6dde1837eb09b0" candidate_uuid: "6ffa6bed5f934e7cb4870a07766cb254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" is_pre_election: true
I20260619 14:59:40.391497 24717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf63a9d05194a9f8a6dde1837eb09b0" candidate_uuid: "6ffa6bed5f934e7cb4870a07766cb254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a00b83581344e6dbbe065a6ffb579db" is_pre_election: true
I20260619 14:59:40.391764 24449 raft_consensus.cc:2454] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ffa6bed5f934e7cb4870a07766cb254 in term 0.
I20260619 14:59:40.391780 24717 raft_consensus.cc:2454] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ffa6bed5f934e7cb4870a07766cb254 in term 0.
I20260619 14:59:40.392198 24518 leader_election.cc:302] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc, 6ffa6bed5f934e7cb4870a07766cb254; no voters:
I20260619 14:59:40.392467 24787 raft_consensus.cc:2781] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:59:40.392547 24787 raft_consensus.cc:482] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:59:40.392589 24787 raft_consensus.cc:3037] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:40.393628 24787 raft_consensus.cc:504] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.393949 24787 leader_election.cc:288] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [CANDIDATE]: Term 1 election: Requested vote from peers 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
I20260619 14:59:40.394289 24449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf63a9d05194a9f8a6dde1837eb09b0" candidate_uuid: "6ffa6bed5f934e7cb4870a07766cb254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dbef50bf0444769b9bd4e2eda487dfc"
I20260619 14:59:40.394404 24449 raft_consensus.cc:3037] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:40.394381 24717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf63a9d05194a9f8a6dde1837eb09b0" candidate_uuid: "6ffa6bed5f934e7cb4870a07766cb254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a00b83581344e6dbbe065a6ffb579db"
I20260619 14:59:40.394537 24717 raft_consensus.cc:3037] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:40.395423 24449 raft_consensus.cc:2454] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ffa6bed5f934e7cb4870a07766cb254 in term 1.
I20260619 14:59:40.395588 24717 raft_consensus.cc:2454] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ffa6bed5f934e7cb4870a07766cb254 in term 1.
I20260619 14:59:40.395687 24518 leader_election.cc:302] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc, 6ffa6bed5f934e7cb4870a07766cb254; no voters:
I20260619 14:59:40.395854 24787 raft_consensus.cc:2781] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:40.396046 24787 raft_consensus.cc:686] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 LEADER]: Becoming Leader. State: Replica: 6ffa6bed5f934e7cb4870a07766cb254, State: Running, Role: LEADER
I20260619 14:59:40.396595 24787 consensus_queue.cc:237] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.399571 24302 catalog_manager.cc:5495] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130). New cstate: current_term: 1 leader_uuid: "6ffa6bed5f934e7cb4870a07766cb254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } health_report { overall_health: HEALTHY } } }
I20260619 14:59:40.410913 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.407921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.10.71.130:33199)
0619 14:59:40.408074 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.408078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.408091 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:59:40.408213 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.408216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.408259 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.408405 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:59:40.408653 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.409088 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.409769 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.409959 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.410026 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.410310 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:59:40.410652 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:59:40.410749 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.410795 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.410834 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:59:40.416102 24449 raft_consensus.cc:1261] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Refusing update from remote peer 6ffa6bed5f934e7cb4870a07766cb254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:40.416109 24717 raft_consensus.cc:1261] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Refusing update from remote peer 6ffa6bed5f934e7cb4870a07766cb254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:40.416846 24787 consensus_queue.cc:1035] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:40.417026 24790 consensus_queue.cc:1035] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:59:40.418810 24764 tablet.cc:2369] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:40.423054 24794 mvcc.cc:205] Tried to move back new op lower bound from 7298585314977947648 to 7298585314906726400. Current Snapshot: MvccSnapshot[applied={T|T < 7298585314977947648}]
I20260619 14:59:40.427531 24795 mvcc.cc:205] Tried to move back new op lower bound from 7298585314977947648 to 7298585314906726400. Current Snapshot: MvccSnapshot[applied={T|T < 7298585314977947648}]
I20260619 14:59:40.430274 24303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:40.430404 24303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:40.432554 24303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Sending DeleteTablet for 3 replicas of tablet 1cf63a9d05194a9f8a6dde1837eb09b0
I20260619 14:59:40.433178 24429 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:40 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:48126
I20260619 14:59:40.433218 24563 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:40 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:46812
I20260619 14:59:40.433318 24697 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:40 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:49954
I20260619 14:59:40.433599 24803 tablet_replica.cc:330] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: stopping tablet replica
I20260619 14:59:40.433656 24804 tablet_replica.cc:330] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: stopping tablet replica
I20260619 14:59:40.433790 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24366
I20260619 14:59:40.433887 24804 raft_consensus.cc:2229] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:40.433887 24803 raft_consensus.cc:2229] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:59:40.434190 24804 raft_consensus.cc:2258] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:40.434190 24803 raft_consensus.cc:2258] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:40.434496 24805 tablet_replica.cc:330] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: stopping tablet replica
I20260619 14:59:40.434759 24805 raft_consensus.cc:2229] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:40.435005 24805 raft_consensus.cc:2258] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:40.435590 24803 ts_tablet_manager.cc:1905] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:40.436556 24805 ts_tablet_manager.cc:1905] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:40.438198 24803 ts_tablet_manager.cc:1918] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:40.438314 24803 log.cc:1196] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/1cf63a9d05194a9f8a6dde1837eb09b0
I20260619 14:59:40.438614 24803 ts_tablet_manager.cc:1939] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting consensus metadata
I20260619 14:59:40.439102 24805 ts_tablet_manager.cc:1918] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:40.439214 24805 log.cc:1196] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/1cf63a9d05194a9f8a6dde1837eb09b0
I20260619 14:59:40.439561 24805 ts_tablet_manager.cc:1939] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 8a00b83581344e6dbbe065a6ffb579db: Deleting consensus metadata
I20260619 14:59:40.439757 24290 catalog_manager.cc:4841] TS 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199): tablet 1cf63a9d05194a9f8a6dde1837eb09b0 (table test-table [id=f036c68c9abd4a148ab90aefcd33f032]) successfully deleted
I20260619 14:59:40.440666 24289 catalog_manager.cc:4841] TS 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): tablet 1cf63a9d05194a9f8a6dde1837eb09b0 (table test-table [id=f036c68c9abd4a148ab90aefcd33f032]) successfully deleted
W20260619 14:59:40.442463 24290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:45727 (error 108)
I20260619 14:59:40.442642 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24500
I20260619 14:59:40.443727 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.443490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:45727 (local address 127.0.0.1:48130)
0619 14:59:40.443609 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.443669 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:59:40.443934 24290 catalog_manager.cc:4584] TS 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cf63a9d05194a9f8a6dde1837eb09b0: Network error: Client connection negotiation failed: client connection to 127.10.71.129:45727: connect: Connection refused (error 111)
I20260619 14:59:40.448794 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24634
I20260619 14:59:40.454159 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24272
I20260619 14:59:40.459661 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42951
--webserver_interface=127.10.71.190
--webserver_port=46529
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:40.562660 24807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:40.562916 24807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:40.563010 24807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:40.566757 24807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:40.566844 24807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:40.566888 24807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:40.566949 24807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:40.567001 24807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:40.571868 24807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:42951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:40.572846 24807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:40.573737 24807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:59:40.586421 24807 server_base.cc:1034] running on GCE node
W20260619 14:59:40.586362 24813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.586362 24814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.586366 24816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:40.587277 24807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:40.588349 24807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:40.589521 24807 hybrid_clock.cc:648] HybridClock initialized: now 1781881180589491 us; error 42 us; skew 500 ppm
Jun 19 14:59:40 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:40.591980 24807 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:40.592814 24807 webserver.cc:460] Webserver started at http://127.10.71.190:46529/ using document root <none> and password file <none>
I20260619 14:59:40.593333 24807 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:40.593427 24807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:40.597107 24807 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:59:40.599097 24823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:40.600263 24807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:40.600395 24807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "63d8ecf3d62147618934619f1449ef60"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "5e6e5616a118cca594e66d382f47e984"
server_key_iv: "6c07dd314275874ec499df0cfd1a9f03"
server_key_version: "encryptionkey@0"
I20260619 14:59:40.601006 24807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:40.617813 24807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:40.620443 24807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:40.620604 24807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:40.627401 24807 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:42951
I20260619 14:59:40.627684 24875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:42951 every 8 connection(s)
I20260619 14:59:40.628518 24807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:40.632700 24876 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:59:40.634303 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap starting.
I20260619 14:59:40.635191 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24807
I20260619 14:59:40.635637 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:45727
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:42951
--webserver_port=36335
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:40.650920 24876 log.cc:824] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:40.660631 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:59:40.661005 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap complete.
I20260619 14:59:40.665050 24876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:40.665482 24876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d8ecf3d62147618934619f1449ef60, State: Initialized, Role: FOLLOWER
I20260619 14:59:40.665975 24876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:40.666128 24876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:40.666218 24876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:40.666333 24876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:59:40.668157 24876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:40.668489 24876 leader_election.cc:302] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d8ecf3d62147618934619f1449ef60; no voters:
I20260619 14:59:40.668800 24876 leader_election.cc:288] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:59:40.668875 24880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:59:40.669113 24880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 LEADER]: Becoming Leader. State: Replica: 63d8ecf3d62147618934619f1449ef60, State: Running, Role: LEADER
I20260619 14:59:40.669485 24880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:40.669904 24876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:40.670678 24881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "63d8ecf3d62147618934619f1449ef60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } } }
I20260619 14:59:40.670805 24881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:40.671422 24882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d8ecf3d62147618934619f1449ef60. Latest consensus state: current_term: 2 leader_uuid: "63d8ecf3d62147618934619f1449ef60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } } }
I20260619 14:59:40.671559 24882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:40.674541 24896 catalog_manager.cc:1259] Loaded cluster ID: 88307da44c244989a384ce0affa49aac
I20260619 14:59:40.674597 24896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: loading cluster ID for follower catalog manager: success
W20260619 14:59:40.675490 24896 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260619 14:59:40.675570 24896 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260619 14:59:40.675637 24889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:40.677696 24889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:40.678155 24889 catalog_manager.cc:1259] Loaded cluster ID: 88307da44c244989a384ce0affa49aac
I20260619 14:59:40.678210 24889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:40.679286 24889 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:59:40.683883 24889 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:59:40.689090 24889 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:40.689776 24889 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Loaded TSK: 0
I20260619 14:59:40.690905 24889 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:59:40.743737 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:40.743979 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:40.744057 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:40.747947 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:40.748028 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:40.748126 24878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:59:40.753163 24878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:45727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=36335
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:40.754043 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:40.754944 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:40.764724 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.764732 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.764819 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:40.765018 24878 server_base.cc:1034] running on GCE node
I20260619 14:59:40.765437 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:40.766065 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:40.767237 24878 hybrid_clock.cc:648] HybridClock initialized: now 1781881180767217 us; error 36 us; skew 500 ppm
Jun 19 14:59:40 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:40.769862 24878 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:59:40.770816 24878 webserver.cc:460] Webserver started at http://127.10.71.129:36335/ using document root <none> and password file <none>
I20260619 14:59:40.771371 24878 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:40.771440 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:40.775126 24878 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:40.777112 24913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:40.778308 24878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:59:40.778390 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "3dbef50bf0444769b9bd4e2eda487dfc"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "463917dec6a7a021a22042249b9cb24a"
server_key_iv: "0adcdad873d9358a7f96fe3cea96aad9"
server_key_version: "encryptionkey@0"
I20260619 14:59:40.778882 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:40.789255 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:40.791998 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:40.792176 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:40.792727 24878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:40.793771 24920 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:59:40.797569 24878 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:59:40.797626 24878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 14:59:40.797685 24878 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:59:40.799158 24878 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:59:40.799226 24878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:59:40.799444 24920 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap starting.
I20260619 14:59:40.809749 24878 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:45727
I20260619 14:59:40.809792 25027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:45727 every 8 connection(s)
I20260619 14:59:40.810920 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:59:40.820569 24920 log.cc:824] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:59:40 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:40.826047 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 24878
I20260619 14:59:40.826537 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:33199
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:42951
--webserver_port=39471
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:40.831892 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.813366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:54345)
0619 14:59:40.815541 (+ 2175us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.815612 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.816617 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.818626 (+ 2009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.818634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.819073 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:40.819623 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.819640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.821453 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.821456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.822705 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.822715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.823706 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.823711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.823941 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.824635 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:40.824655 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:40.826416 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:40.828868 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:40.828872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:40.828883 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:40.829195 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:40.829551 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:40.829554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:40.829556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:40.829733 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:40.830377 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:40.830382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:40.830629 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.830799 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.831063 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"thread_start_us":132,"threads_started":1}
I20260619 14:59:40.833280 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.814610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:54345 (local address 127.10.71.190:42951)
0619 14:59:40.817142 (+ 2532us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.817159 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.817194 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:59:40.817420 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.817434 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.817768 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.818136 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:40.819824 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.821302 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.822832 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.823493 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.823705 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.826636 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:40.826642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:40.826657 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:40.826688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:40.828746 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:40.829307 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:40.829312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:40.829317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:40.829437 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:40.829841 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:40.829843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:40.829845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:40.830213 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:40.830329 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.832257 (+ 1928us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.832551 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":105,"threads_started":1}
I20260619 14:59:40.834143 24920 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:59:40.834514 24920 tablet_bootstrap.cc:492] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap complete.
I20260619 14:59:40.835426 25028 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42951
I20260619 14:59:40.835650 25028 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:40.835745 24920 ts_tablet_manager.cc:1397] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.011s
I20260619 14:59:40.836098 25028 heartbeater.cc:511] Master 127.10.71.190:42951 requested a full tablet report, sending...
I20260619 14:59:40.838160 24920 raft_consensus.cc:348] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.838542 24920 raft_consensus.cc:729] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbef50bf0444769b9bd4e2eda487dfc, State: Initialized, Role: FOLLOWER
I20260619 14:59:40.838637 24840 ts_manager.cc:194] Registered new tserver with Master: 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727)
I20260619 14:59:40.839021 24920 consensus_queue.cc:260] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } }
I20260619 14:59:40.839828 24920 ts_tablet_manager.cc:1428] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:59:40.842212 24840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:54345
I20260619 14:59:40.844197 25028 heartbeater.cc:503] Master 127.10.71.190:42951 was elected leader, sending a full tablet report...
I20260619 14:59:40.849004 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.841536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:45727 (local address 127.0.0.1:48136)
0619 14:59:40.844780 (+ 3244us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:40.844820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:40.844964 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:40.845394 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:40.845399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:40.845419 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:40.845524 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.845530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.846874 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.846877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.847640 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:40.847647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.848656 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.848659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:40.848735 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.848760 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:59:40.848836 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:59:40.848916 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3097,"thread_start_us":102,"threads_started":1}
I20260619 14:59:40.849082 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:40.841563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.10.71.129:45727)
0619 14:59:40.844533 (+ 2970us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:40.844551 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:40.844902 (+ 351us) server_negotiation.cc:407] Connection header received
0619 14:59:40.845098 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:40.845105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:40.845241 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:40.845399 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:40.845652 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.846765 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.847778 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:40.848551 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:40.848656 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:40.848766 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:40.848872 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 14:59:40.848951 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2877,"thread_start_us":91,"threads_started":1}
I20260619 14:59:40.849941 24954 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf63a9d05194a9f8a6dde1837eb09b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:40 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:48136
I20260619 14:59:40.850675 25037 tablet_replica.cc:330] stopping tablet replica
I20260619 14:59:40.850956 25037 raft_consensus.cc:2229] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:40.851172 25037 raft_consensus.cc:2258] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:40.852054 25037 ts_tablet_manager.cc:1905] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:40.856068 25037 ts_tablet_manager.cc:1918] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:40.856159 25037 log.cc:1196] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/1cf63a9d05194a9f8a6dde1837eb09b0
I20260619 14:59:40.856455 25037 ts_tablet_manager.cc:1939] T 1cf63a9d05194a9f8a6dde1837eb09b0 P 3dbef50bf0444769b9bd4e2eda487dfc: Deleting consensus metadata
I20260619 14:59:40.858193 24826 catalog_manager.cc:4841] TS 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727): tablet 1cf63a9d05194a9f8a6dde1837eb09b0 (table test-table [id=f036c68c9abd4a148ab90aefcd33f032]) successfully deleted
W20260619 14:59:40.936636 25033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:40.936843 25033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:40.936882 25033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:40.940773 25033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:40.940817 25033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:40.940898 25033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:59:40.945870 25033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:33199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=39471
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:40.946784 25033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:40.947700 25033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:40.956545 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.956557 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:40.956552 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:40.956785 25033 server_base.cc:1034] running on GCE node
I20260619 14:59:40.957194 25033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:40.957831 25033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:40.959033 25033 hybrid_clock.cc:648] HybridClock initialized: now 1781881180959029 us; error 34 us; skew 500 ppm
Jun 19 14:59:40 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:40.961853 25033 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:59:40.962639 25033 webserver.cc:460] Webserver started at http://127.10.71.130:39471/ using document root <none> and password file <none>
I20260619 14:59:40.963176 25033 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:40.963284 25033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:40.966917 25033 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:59:40.969004 25052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:40.970270 25033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:40.970395 25033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "6ffa6bed5f934e7cb4870a07766cb254"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "dfd364a2d33340cf1d74cac01c1eb468"
server_key_iv: "e3c5ffa809b1253b18fb1ca0c7a0fcf1"
server_key_version: "encryptionkey@0"
I20260619 14:59:40.970990 25033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:41.007524 25033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:41.010356 25033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:41.010535 25033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:41.011070 25033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:41.011788 25033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:41.011878 25033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.011965 25033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:41.012014 25033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.020340 25033 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:33199
I20260619 14:59:41.020366 25165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:33199 every 8 connection(s)
I20260619 14:59:41.021196 25033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:59:41.024225 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25033
I20260619 14:59:41.024740 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:36035
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:42951
--webserver_port=33865
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:59:41 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881180, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:41.035442 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.022691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:52981 (local address 127.10.71.190:42951)
0619 14:59:41.022815 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.022818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.023603 (+ 785us) server_negotiation.cc:407] Connection header received
0619 14:59:41.025421 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.025423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.025473 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.025543 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:41.026772 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.027617 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.028811 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.029255 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.029375 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.031676 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:41.031681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:41.031684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:41.031714 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:41.033357 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:41.033853 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:41.033859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:41.033861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:41.033959 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:41.034279 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:41.034284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:41.034287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:41.034537 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:41.034656 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.035065 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.035216 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:59:41.035923 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.023047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:52981)
0619 14:59:41.023459 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.023498 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.025199 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.025721 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:41.025731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:41.026193 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:41.026631 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.026644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.027753 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.027756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.028698 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.028706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.029382 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.029385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.029506 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.030017 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:41.030031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:41.031497 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:41.033509 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:41.033514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:41.033526 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:41.033752 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:41.034072 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:41.034075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:41.034076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:41.034189 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:41.034638 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:41.034642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:41.034832 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 14:59:41.034968 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 14:59:41.035196 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
I20260619 14:59:41.036972 25166 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42951
I20260619 14:59:41.037189 25166 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:41.037658 25166 heartbeater.cc:511] Master 127.10.71.190:42951 requested a full tablet report, sending...
I20260619 14:59:41.038708 24840 ts_manager.cc:194] Registered new tserver with Master: 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199)
I20260619 14:59:41.039224 24840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:52981
W20260619 14:59:41.130434 25169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:41.130645 25169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:41.130681 25169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:41.134446 25169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:41.134492 25169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:41.134584 25169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:59:41.139266 25169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:36035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=33865
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:41.140172 25169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:41.141105 25169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:41.149856 25177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:41.149981 25169 server_base.cc:1034] running on GCE node
W20260619 14:59:41.149871 25175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:41.149906 25174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:41.150514 25169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:41.151082 25169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:41.152268 25169 hybrid_clock.cc:648] HybridClock initialized: now 1781881181152255 us; error 24 us; skew 500 ppm
Jun 19 14:59:41 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881181, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:41.154870 25169 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:59:41.155679 25169 webserver.cc:460] Webserver started at http://127.10.71.131:33865/ using document root <none> and password file <none>
I20260619 14:59:41.156190 25169 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:41.156288 25169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:41.159785 25169 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:59:41.161831 25184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.162956 25169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:59:41.163072 25169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "8a00b83581344e6dbbe065a6ffb579db"
format_stamp: "Formatted at 2026-06-19 14:59:40 on dist-test-slave-3qdh"
server_key: "5c192e9e3da4ebca9ad082e20f212c7b"
server_key_iv: "916f76d234d99b4e0556f4d081354137"
server_key_version: "encryptionkey@0"
I20260619 14:59:41.163544 25169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:41.179411 25169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:41.182830 25169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:41.183043 25169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:41.183658 25169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:41.184486 25169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:41.184552 25169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.184623 25169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:41.184672 25169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.193446 25169 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:36035
I20260619 14:59:41.193488 25297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:36035 every 8 connection(s)
I20260619 14:59:41.194438 25169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:59:41.200788 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25169
Jun 19 14:59:41 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881181, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:41.207756 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.196093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:39125 (local address 127.10.71.190:42951)
0619 14:59:41.196213 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.196217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.196985 (+ 768us) server_negotiation.cc:407] Connection header received
0619 14:59:41.198262 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.198266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.198323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.198420 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:41.199499 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.200072 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.201439 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.201734 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.201841 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.204135 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:41.204138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:41.204141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:41.204164 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:41.205763 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:41.206287 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:41.206292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:41.206294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:41.206378 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:41.206696 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:41.206701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:41.206703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:41.206874 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:41.206976 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.207418 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.207577 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:59:41.208485 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.196453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:39125)
0619 14:59:41.196849 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.196887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.198054 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.198585 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:41.198591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:41.198904 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:41.199358 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.199370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.200188 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.200191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.201306 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.201318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.201873 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.201877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.202044 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.202586 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:41.202600 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:41.203960 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:41.205884 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:41.205890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:41.205905 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:41.206184 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:41.206467 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:41.206472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:41.206474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:41.206604 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:41.206974 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:41.206979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:41.207200 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:59:41.207380 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:59:41.207652 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260619 14:59:41.209515 25298 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42951
I20260619 14:59:41.209740 25298 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:41.210260 25298 heartbeater.cc:511] Master 127.10.71.190:42951 requested a full tablet report, sending...
I20260619 14:59:41.211236 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.201401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.10.71.190:42951)
0619 14:59:41.201617 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.201621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.201846 (+ 225us) server_negotiation.cc:407] Connection header received
0619 14:59:41.201985 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.201988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.202036 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.202129 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:41.202820 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.203314 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.204321 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.204646 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.204752 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.206406 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:41.206410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:41.206413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:41.206447 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:41.208140 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:41.208638 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:41.208641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:41.208643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:41.208726 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:41.209051 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:41.209057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:41.209059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:41.209265 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:41.209523 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.210969 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.211076 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260619 14:59:41.211277 24840 ts_manager.cc:194] Registered new tserver with Master: 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
I20260619 14:59:41.211841 24840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:39125
I20260619 14:59:41.212978 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:59:41.217034 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.213553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.10.71.190:42951)
0619 14:59:41.213642 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.213647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.213832 (+ 185us) server_negotiation.cc:407] Connection header received
0619 14:59:41.213968 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.213971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.214054 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.214181 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:59:41.214406 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.214880 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.215720 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.215929 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.216021 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.216360 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:59:41.216687 (+ 327us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:59:41.216810 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.216870 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.216921 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 14:59:41.217595 24840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:59:41.220674 24840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:59:41.227339 24954 tablet_service.cc:1467] Processing CreateTablet for tablet 97dbbb978aef42cc9109c952917f6743 (DEFAULT_TABLE table=test-table [id=9a41192c61b743e08e71ef2c11d5705b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:41.227964 24954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dbbb978aef42cc9109c952917f6743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:41.230643 24920 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap starting.
I20260619 14:59:41.231408 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.226340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:33199 (local address 127.0.0.1:46834)
0619 14:59:41.226480 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.226494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.226556 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.227603 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:41.227606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:41.227622 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:41.227713 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.227719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.229249 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.229252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.230033 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.230042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.231028 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.231031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.231142 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.231164 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:59:41.231225 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:59:41.231297 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:59:41.231613 24920 tablet_bootstrap.cc:654] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:41.231756 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.226519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.10.71.130:33199)
0619 14:59:41.227106 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.227117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.227155 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:59:41.227230 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.227235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.227429 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.227877 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:41.228041 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.229082 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.230164 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.230895 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.231025 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.231188 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.231271 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.231366 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":127,"threads_started":1}
I20260619 14:59:41.232169 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.226753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36035 (local address 127.0.0.1:49970)
0619 14:59:41.226978 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.226993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.227080 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.227816 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:41.227820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:41.227838 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:41.227940 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.227950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.229543 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.229546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.230550 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.230560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.231771 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.231780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.231903 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.231924 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:59:41.232004 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:59:41.232053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260619 14:59:41.232360 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.227020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.10.71.131:36035)
0619 14:59:41.227381 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.227389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.227407 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:59:41.227461 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.227464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.227663 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.227846 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:41.228131 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.229401 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.230693 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.231435 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.231539 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.231709 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.232073 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.232181 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260619 14:59:41.232908 24920 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: No bootstrap required, opened a new log
I20260619 14:59:41.233031 24920 ts_tablet_manager.cc:1397] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:59:41.233561 24920 raft_consensus.cc:348] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.233673 24920 raft_consensus.cc:374] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:41.233706 24920 raft_consensus.cc:729] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbef50bf0444769b9bd4e2eda487dfc, State: Initialized, Role: FOLLOWER
I20260619 14:59:41.233837 24920 consensus_queue.cc:260] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.234098 24920 ts_tablet_manager.cc:1428] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:59:41.234930 25100 tablet_service.cc:1467] Processing CreateTablet for tablet 97dbbb978aef42cc9109c952917f6743 (DEFAULT_TABLE table=test-table [id=9a41192c61b743e08e71ef2c11d5705b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:41.234936 25232 tablet_service.cc:1467] Processing CreateTablet for tablet 97dbbb978aef42cc9109c952917f6743 (DEFAULT_TABLE table=test-table [id=9a41192c61b743e08e71ef2c11d5705b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:41.235881 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dbbb978aef42cc9109c952917f6743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:41.235929 25232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dbbb978aef42cc9109c952917f6743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:41.243187 25311 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Bootstrap starting.
I20260619 14:59:41.244424 25312 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Bootstrap starting.
I20260619 14:59:41.246083 25311 tablet_bootstrap.cc:654] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:41.246820 25311 log.cc:824] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:41.247387 25312 tablet_bootstrap.cc:654] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:41.248327 25312 log.cc:824] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:41.249533 25311 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: No bootstrap required, opened a new log
I20260619 14:59:41.249729 25311 ts_tablet_manager.cc:1397] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:59:41.250114 25312 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: No bootstrap required, opened a new log
I20260619 14:59:41.250312 25312 ts_tablet_manager.cc:1397] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:59:41.252732 25311 raft_consensus.cc:348] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.252954 25311 raft_consensus.cc:374] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:41.253012 25311 raft_consensus.cc:729] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a00b83581344e6dbbe065a6ffb579db, State: Initialized, Role: FOLLOWER
I20260619 14:59:41.253298 25312 raft_consensus.cc:348] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.253500 25312 raft_consensus.cc:374] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:41.253435 25311 consensus_queue.cc:260] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.253579 25312 raft_consensus.cc:729] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ffa6bed5f934e7cb4870a07766cb254, State: Initialized, Role: FOLLOWER
I20260619 14:59:41.254019 25312 consensus_queue.cc:260] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.254102 25298 heartbeater.cc:503] Master 127.10.71.190:42951 was elected leader, sending a full tablet report...
I20260619 14:59:41.254474 25311 ts_tablet_manager.cc:1428] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:59:41.254722 25166 heartbeater.cc:503] Master 127.10.71.190:42951 was elected leader, sending a full tablet report...
I20260619 14:59:41.254959 25312 ts_tablet_manager.cc:1428] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 14:59:41.271974 25167 tablet.cc:2369] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:59:41.315649 25029 tablet.cc:2369] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:41.443789 25034 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:41.443981 25034 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.445076 25034 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
W20260619 14:59:41.445398 25299 tablet.cc:2369] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:41.448665 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.445833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:37501 (local address 127.10.71.131:36035)
0619 14:59:41.445969 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.445976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.445992 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:59:41.446046 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.446048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.446083 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.446172 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:41.446405 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.446828 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.447709 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.448081 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.448194 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.448240 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.448487 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.448548 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:59:41.448675 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.445413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:41071 (local address 127.10.71.130:33199)
0619 14:59:41.445554 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.445559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.445679 (+ 120us) server_negotiation.cc:407] Connection header received
0619 14:59:41.445857 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.445861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.445928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.445970 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:41.446308 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.446762 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.447703 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.448092 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.448210 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.448257 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.448496 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.448557 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:59:41.448685 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.445254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:33199 (local address 127.10.71.129:41071)
0619 14:59:41.445596 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.445611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.445716 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.446092 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:41.446094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:41.446107 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:41.446185 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.446189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.446868 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.446871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.447592 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.447600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.448204 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.448208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.448386 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.448404 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:59:41.448482 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:59:41.448557 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":47,"thread_start_us":96,"threads_started":1}
I20260619 14:59:41.448693 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.445290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36035 (local address 127.10.71.129:37501)
0619 14:59:41.445656 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.445669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.445754 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.446183 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:41.446186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:41.446202 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:41.446306 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.446312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.446923 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.446927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.447594 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:41.447603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.448189 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.448193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:41.448375 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.448394 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:59:41.448472 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:59:41.448548 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":76,"thread_start_us":72,"threads_started":1}
I20260619 14:59:41.449329 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dbbb978aef42cc9109c952917f6743" candidate_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ffa6bed5f934e7cb4870a07766cb254" is_pre_election: true
I20260619 14:59:41.449326 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dbbb978aef42cc9109c952917f6743" candidate_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a00b83581344e6dbbe065a6ffb579db" is_pre_election: true
I20260619 14:59:41.449602 25120 raft_consensus.cc:2454] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbef50bf0444769b9bd4e2eda487dfc in term 0.
I20260619 14:59:41.449625 25252 raft_consensus.cc:2454] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbef50bf0444769b9bd4e2eda487dfc in term 0.
I20260619 14:59:41.450012 24916 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc, 6ffa6bed5f934e7cb4870a07766cb254; no voters:
I20260619 14:59:41.450263 25034 raft_consensus.cc:2781] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:59:41.450341 25034 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:59:41.450381 25034 raft_consensus.cc:3037] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:41.451337 25034 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.451668 25034 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 1 election: Requested vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
I20260619 14:59:41.452009 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dbbb978aef42cc9109c952917f6743" candidate_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ffa6bed5f934e7cb4870a07766cb254"
I20260619 14:59:41.452126 25120 raft_consensus.cc:3037] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:41.452083 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dbbb978aef42cc9109c952917f6743" candidate_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a00b83581344e6dbbe065a6ffb579db"
I20260619 14:59:41.452243 25252 raft_consensus.cc:3037] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:41.453065 25120 raft_consensus.cc:2454] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbef50bf0444769b9bd4e2eda487dfc in term 1.
I20260619 14:59:41.453223 25252 raft_consensus.cc:2454] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbef50bf0444769b9bd4e2eda487dfc in term 1.
I20260619 14:59:41.453341 24916 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc, 6ffa6bed5f934e7cb4870a07766cb254; no voters:
I20260619 14:59:41.453559 25034 raft_consensus.cc:2781] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:41.453720 25034 raft_consensus.cc:686] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 LEADER]: Becoming Leader. State: Replica: 3dbef50bf0444769b9bd4e2eda487dfc, State: Running, Role: LEADER
I20260619 14:59:41.453967 25034 consensus_queue.cc:237] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.456779 24839 catalog_manager.cc:5495] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129). New cstate: current_term: 1 leader_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } health_report { overall_health: HEALTHY } } }
I20260619 14:59:41.477259 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.473845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.10.71.129:45727)
0619 14:59:41.474104 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:41.474114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:41.474131 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:59:41.474195 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:41.474201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:41.474244 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:41.474360 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:59:41.474594 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.475095 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.475906 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:41.476115 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:41.476208 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:41.476543 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:59:41.476893 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:59:41.477027 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:41.477074 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:59:41.477118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260619 14:59:41.480506 25252 raft_consensus.cc:1261] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Refusing update from remote peer 3dbef50bf0444769b9bd4e2eda487dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:41.480521 25120 raft_consensus.cc:1261] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Refusing update from remote peer 3dbef50bf0444769b9bd4e2eda487dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:41.481096 25319 consensus_queue.cc:1035] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:41.481277 25034 consensus_queue.cc:1035] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:41.486783 25323 mvcc.cc:205] Tried to move back new op lower bound from 7298585319338500096 to 7298585319237697536. Current Snapshot: MvccSnapshot[applied={T|T < 7298585319338500096}]
I20260619 14:59:41.495368 24840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:41.495476 24840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:41.497325 24840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Sending DeleteTablet for 3 replicas of tablet 97dbbb978aef42cc9109c952917f6743
I20260619 14:59:41.497924 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dbbb978aef42cc9109c952917f6743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:41 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:46834
I20260619 14:59:41.497907 25232 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dbbb978aef42cc9109c952917f6743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:41 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:49970
I20260619 14:59:41.497990 24954 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dbbb978aef42cc9109c952917f6743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:41 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:48136
I20260619 14:59:41.498351 25334 tablet_replica.cc:330] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: stopping tablet replica
I20260619 14:59:41.498452 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24878
I20260619 14:59:41.500466 25335 tablet_replica.cc:330] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: stopping tablet replica
I20260619 14:59:41.500694 25335 raft_consensus.cc:2229] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:41.500948 25335 raft_consensus.cc:2258] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:41.501291 25334 raft_consensus.cc:2229] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:41.501526 25334 raft_consensus.cc:2258] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:41.502301 25335 ts_tablet_manager.cc:1905] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:41.502663 25334 ts_tablet_manager.cc:1905] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:59:41.503703 24826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:45727 (error 108)
I20260619 14:59:41.504747 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.504464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:45727 (local address 127.0.0.1:48160)
0619 14:59:41.504584 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.504655 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:59:41.504769 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 25033
I20260619 14:59:41.504834 25335 ts_tablet_manager.cc:1918] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:41.504925 25335 log.cc:1196] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/97dbbb978aef42cc9109c952917f6743
I20260619 14:59:41.505100 25334 ts_tablet_manager.cc:1918] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:41.505201 25334 log.cc:1196] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/97dbbb978aef42cc9109c952917f6743
I20260619 14:59:41.505476 25334 ts_tablet_manager.cc:1939] T 97dbbb978aef42cc9109c952917f6743 P 8a00b83581344e6dbbe065a6ffb579db: Deleting consensus metadata
I20260619 14:59:41.506332 24825 catalog_manager.cc:4841] TS 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): tablet 97dbbb978aef42cc9109c952917f6743 (table test-table [id=9a41192c61b743e08e71ef2c11d5705b]) successfully deleted
W20260619 14:59:41.508450 24826 catalog_manager.cc:4584] TS 3dbef50bf0444769b9bd4e2eda487dfc (127.10.71.129:45727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97dbbb978aef42cc9109c952917f6743: Network error: Client connection negotiation failed: client connection to 127.10.71.129:45727: connect: Connection refused (error 111)
I20260619 14:59:41.511363 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 25169
I20260619 14:59:41.511785 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.511499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:33199 (local address 127.0.0.1:46844)
0619 14:59:41.511665 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.511715 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.130:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 14:59:41.516559 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 24807
I20260619 14:59:41.522156 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42951
--webserver_interface=127.10.71.190
--webserver_port=46529
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:41.622758 25337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:41.622962 25337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:41.622998 25337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:41.626842 25337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:41.626888 25337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:41.626905 25337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:41.626921 25337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:41.626977 25337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:41.631673 25337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:42951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:41.632493 25337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:41.633327 25337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:41.640877 25342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:41.640853 25343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:41.640887 25337 server_base.cc:1034] running on GCE node
W20260619 14:59:41.640859 25345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:41.641587 25337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:41.642549 25337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:41.643716 25337 hybrid_clock.cc:648] HybridClock initialized: now 1781881181643708 us; error 24 us; skew 500 ppm
Jun 19 14:59:41 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881181, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:41.656824 25337 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:41.657636 25337 webserver.cc:460] Webserver started at http://127.10.71.190:46529/ using document root <none> and password file <none>
I20260619 14:59:41.658154 25337 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:41.658242 25337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:41.661815 25337 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:59:41.663705 25352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.664894 25337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:41.665042 25337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "63d8ecf3d62147618934619f1449ef60"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "5e6e5616a118cca594e66d382f47e984"
server_key_iv: "6c07dd314275874ec499df0cfd1a9f03"
server_key_version: "encryptionkey@0"
I20260619 14:59:41.665534 25337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:41.685840 25337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:41.688449 25337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:41.688633 25337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:41.696239 25337 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:42951
I20260619 14:59:41.696801 25404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:42951 every 8 connection(s)
I20260619 14:59:41.697570 25337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:41.703209 25405 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:59:41.705772 25405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap starting.
I20260619 14:59:41.707401 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25337
I20260619 14:59:41.707974 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:45727
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:42951
--webserver_port=36335
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:41.727129 25405 log.cc:824] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:41.738090 25405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:59:41.738401 25405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60: Bootstrap complete.
I20260619 14:59:41.742028 25405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:41.742369 25405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d8ecf3d62147618934619f1449ef60, State: Initialized, Role: FOLLOWER
I20260619 14:59:41.742839 25405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:41.742974 25405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:41.743067 25405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:41.743170 25405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:59:41.744939 25405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:41.745249 25405 leader_election.cc:302] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d8ecf3d62147618934619f1449ef60; no voters:
I20260619 14:59:41.745532 25405 leader_election.cc:288] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:59:41.745656 25409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:59:41.745860 25409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [term 3 LEADER]: Becoming Leader. State: Replica: 63d8ecf3d62147618934619f1449ef60, State: Running, Role: LEADER
I20260619 14:59:41.746117 25409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } }
I20260619 14:59:41.746575 25405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:41.746956 25409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d8ecf3d62147618934619f1449ef60. Latest consensus state: current_term: 3 leader_uuid: "63d8ecf3d62147618934619f1449ef60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } } }
I20260619 14:59:41.747052 25409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:41.747628 25410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "63d8ecf3d62147618934619f1449ef60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d8ecf3d62147618934619f1449ef60" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42951 } } }
I20260619 14:59:41.747716 25410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d8ecf3d62147618934619f1449ef60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:41.748121 25413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:41.753067 25413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:41.753662 25413 catalog_manager.cc:1259] Loaded cluster ID: 88307da44c244989a384ce0affa49aac
I20260619 14:59:41.753719 25413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:59:41.754667 25413 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c47b3f9ad google::LogMessage::Fail() at ??:0
@ 0x7f9c47b4288a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c47b3f4de google::LogMessage::Flush() at ??:0
@ 0x7f9c47b40a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c5247efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9c5247fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c5247c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9c524c0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9c49d913d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c49d91c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9c49d933f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9c49d82512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c4b3c66db start_thread at ??:0
@ 0x7f9c4663d61f clone at ??:0
W20260619 14:59:41.839883 25407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:41.840094 25407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:41.840166 25407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:41.843770 25407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:41.843847 25407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:41.843964 25407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:59:41.848771 25407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:45727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=36335
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:41.849578 25407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:41.850384 25407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:41.858964 25446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:41.858944 25449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:41.859155 25407 server_base.cc:1034] running on GCE node
W20260619 14:59:41.858989 25447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:41.859591 25407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:41.860211 25407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:41.861372 25407 hybrid_clock.cc:648] HybridClock initialized: now 1781881181861353 us; error 44 us; skew 500 ppm
Jun 19 14:59:41 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881181, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:41.863812 25407 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:59:41.864699 25407 webserver.cc:460] Webserver started at http://127.10.71.129:36335/ using document root <none> and password file <none>
I20260619 14:59:41.865191 25407 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:41.865247 25407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:41.868683 25407 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:59:41.870711 25456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:41.871856 25407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:41.871986 25407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "3dbef50bf0444769b9bd4e2eda487dfc"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "463917dec6a7a021a22042249b9cb24a"
server_key_iv: "0adcdad873d9358a7f96fe3cea96aad9"
server_key_version: "encryptionkey@0"
I20260619 14:59:41.872527 25407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:41.888561 25407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:41.891260 25407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:41.891436 25407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:41.891921 25407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:41.893019 25463 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:59:41.896641 25407 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:59:41.896694 25407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 14:59:41.896734 25407 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:59:41.898162 25407 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:59:41.898213 25407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 14:59:41.898433 25463 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap starting.
I20260619 14:59:41.908171 25407 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:45727
I20260619 14:59:41.908430 25570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:45727 every 8 connection(s)
I20260619 14:59:41.909269 25407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 14:59:41.914788 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25407
I20260619 14:59:41.915252 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:33199
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:42951
--webserver_port=39471
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:59:41.918285 25463 log.cc:824] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:41.926435 25463 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:59:41.926822 25463 tablet_bootstrap.cc:492] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Bootstrap complete.
I20260619 14:59:41.928246 25463 ts_tablet_manager.cc:1397] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260619 14:59:41.931309 25463 raft_consensus.cc:348] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.931838 25463 raft_consensus.cc:729] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbef50bf0444769b9bd4e2eda487dfc, State: Initialized, Role: FOLLOWER
I20260619 14:59:41.932370 25463 consensus_queue.cc:260] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:41.933334 25463 ts_tablet_manager.cc:1428] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260619 14:59:41.955865 25573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:59:41.911308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:43519)
0619 14:59:41.911810 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.911862 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:41.913030 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:41.955216 (+ 42186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":129,"threads_started":1}
W20260619 14:59:41.956373 25571 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:42951 (0 consecutive failures): Network error: Failed to ping master at 127.10.71.190:42951: Client connection negotiation failed: client connection to 127.10.71.190:42951: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:59:41.957129 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.956891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:41287)
0619 14:59:41.957005 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.957077 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:59:41.957864 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:41.957673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:56205)
0619 14:59:41.957774 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:41.957818 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:59:41.958117 25571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:59:42.021637 25574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:42.021847 25574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:42.021885 25574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:42.025673 25574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:42.025717 25574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:42.025764 25574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:59:42.030510 25574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:33199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=39471
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:42.031368 25574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:42.032279 25574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:42.040752 25582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:42.040776 25584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:42.040817 25581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:42.040944 25574 server_base.cc:1034] running on GCE node
I20260619 14:59:42.041384 25574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:42.041940 25574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:42.043123 25574 hybrid_clock.cc:648] HybridClock initialized: now 1781881182043109 us; error 27 us; skew 500 ppm
Jun 19 14:59:42 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881182, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:42.045727 25574 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:59:42.046363 25574 webserver.cc:460] Webserver started at http://127.10.71.130:39471/ using document root <none> and password file <none>
I20260619 14:59:42.046793 25574 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:42.046845 25574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:42.050052 25574 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:59:42.051704 25591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:42.052841 25574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:42.052932 25574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "6ffa6bed5f934e7cb4870a07766cb254"
format_stamp: "Formatted at 2026-06-19 14:59:39 on dist-test-slave-3qdh"
server_key: "dfd364a2d33340cf1d74cac01c1eb468"
server_key_iv: "e3c5ffa809b1253b18fb1ca0c7a0fcf1"
server_key_version: "encryptionkey@0"
I20260619 14:59:42.053371 25574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:42.064106 25574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:42.066943 25574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:42.067078 25574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:42.067498 25574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:42.068470 25598 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 14:59:42.071933 25598 ts_tablet_manager.cc:1852] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 14:59:42.072273 25598 ts_tablet_manager.cc:1905] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:42.075673 25598 ts_tablet_manager.cc:1918] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:42.075904 25598 ts_tablet_manager.cc:1939] T 97dbbb978aef42cc9109c952917f6743 P 6ffa6bed5f934e7cb4870a07766cb254: Deleting consensus metadata
I20260619 14:59:42.076233 25574 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 14:59:42.076303 25574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260619 14:59:42.076411 25574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:42.076462 25574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:42.084645 25574 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:33199
I20260619 14:59:42.084697 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:33199 every 8 connection(s)
I20260619 14:59:42.085384 25574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 14:59:42.088225 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.087101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:39499)
0619 14:59:42.087513 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.087599 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
W20260619 14:59:42.088696 25706 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:42951 (0 consecutive failures): Network error: Failed to ping master at 127.10.71.190:42951: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
I20260619 14:59:42.089228 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.089012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:49837)
0619 14:59:42.089132 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.089178 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:59:42.089825 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.089648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:50585)
0619 14:59:42.089746 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.089782 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:59:42.089958 25706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:59:42.093127 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25574
I20260619 14:59:42.093453 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:36035
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:42951
--webserver_port=33865
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:42.198158 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:42.198412 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:42.198458 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:42.202512 25709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:42.202566 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:42.202625 25709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:59:42.208002 25709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:36035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=33865
--tserver_master_addrs=127.10.71.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:42.208992 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:42.209888 25709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:42.218851 25715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:42.218839 25717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:42.218937 25714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:42.219152 25709 server_base.cc:1034] running on GCE node
I20260619 14:59:42.219468 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:42.220021 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:42.221175 25709 hybrid_clock.cc:648] HybridClock initialized: now 1781881182221147 us; error 38 us; skew 500 ppm
Jun 19 14:59:42 dist-test-slave-3qdh krb5kdc[24259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881182, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:42.223575 25709 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:59:42.224434 25709 webserver.cc:460] Webserver started at http://127.10.71.131:33865/ using document root <none> and password file <none>
I20260619 14:59:42.224936 25709 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:42.224994 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:42.228399 25709 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 14:59:42.230157 25724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:42.231243 25709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:42.231343 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "8a00b83581344e6dbbe065a6ffb579db"
format_stamp: "Formatted at 2026-06-19 14:59:40 on dist-test-slave-3qdh"
server_key: "5c192e9e3da4ebca9ad082e20f212c7b"
server_key_iv: "916f76d234d99b4e0556f4d081354137"
server_key_version: "encryptionkey@0"
I20260619 14:59:42.231794 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:42.249200 25709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:42.251427 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:42.251597 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:42.252063 25709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:42.252760 25709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:42.252820 25709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:42.252887 25709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:42.252924 25709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:42.261276 25709 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:36035
I20260619 14:59:42.261308 25837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:36035 every 8 connection(s)
I20260619 14:59:42.262125 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:59:42.265414 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.263789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:46859)
0619 14:59:42.264192 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.264374 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
W20260619 14:59:42.265856 25838 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:42951 (0 consecutive failures): Network error: Failed to ping master at 127.10.71.190:42951: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
I20260619 14:59:42.266350 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.266139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:52431)
0619 14:59:42.266237 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.266284 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:59:42.266922 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.266748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:53303)
0619 14:59:42.266837 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.266879 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:59:42.267160 25838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:59:42.269215 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25709
I20260619 14:59:42.958918 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:42.958585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:36169)
0619 14:59:42.958788 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:42.958834 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260619 14:59:43.090698 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.090384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:45979)
0619 14:59:43.090576 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:43.090621 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260619 14:59:43.267895 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.267558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:45981)
0619 14:59:43.267753 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:43.267812 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20260619 14:59:43.276086 25845 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:43.276271 25845 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:43.277391 25845 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
Jun 19 14:59:43 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881181, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.131@KRBTEST.COM
Jun 19 14:59:43 dist-test-slave-3qdh krb5kdc[24259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881181, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.130@KRBTEST.COM
I20260619 14:59:43.290872 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.277567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:36035 (local address 127.10.71.129:35035)
0619 14:59:43.277956 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:43.277970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:43.278078 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:43.279378 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:43.279384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:43.279867 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:43.280274 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:43.280287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.281448 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.281451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:43.282281 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:43.282289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.282966 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.282969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:43.283193 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:43.283803 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:43.283818 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:43.285442 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:43.288291 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:43.288315 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:43.288317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:43.288627 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:43.289000 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:43.289003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:43.289005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:43.289224 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:43.289900 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:43.289906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:43.290196 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 14:59:43.290374 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 14:59:43.290715 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":256,"spinlock_wait_cycles":26112,"thread_start_us":80,"threads_started":1}
I20260619 14:59:43.290871 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.277486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:33199 (local address 127.10.71.129:60929)
0619 14:59:43.277798 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:43.277813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:43.277911 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:43.279378 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:43.279384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:43.279867 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:43.280275 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:43.280287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.281453 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.281456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:43.282280 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:43.282290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.282970 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.282974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:43.283188 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:43.283864 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:43.283876 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:43.285762 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:43.288183 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:43.288190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:43.288203 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:43.288625 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:43.289000 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:43.289003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:43.289005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:43.289224 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:43.289900 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:43.289906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:43.290196 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 14:59:43.290374 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 14:59:43.290671 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":189,"spinlock_wait_cycles":24320}
I20260619 14:59:43.290926 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.277758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:35035 (local address 127.10.71.131:36035)
0619 14:59:43.278121 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:43.278145 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:43.278179 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:59:43.278771 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:43.278776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:43.279077 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:43.279341 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:43.280460 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.281313 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.282393 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.282850 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.283020 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:43.285963 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:43.285967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:43.285976 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:43.286024 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:43.288067 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:43.288741 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:43.288747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:43.288753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:43.288902 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:43.289330 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:43.289334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:43.289335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:43.289790 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:43.289912 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:43.290523 (+ 611us) server_negotiation.cc:299] Negotiation successful
0619 14:59:43.290764 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":94,"threads_started":1}
I20260619 14:59:43.291394 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.277807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:60929 (local address 127.10.71.130:33199)
0619 14:59:43.278134 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:43.278144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:43.278175 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:59:43.278862 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:43.278868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:43.279138 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:43.279389 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:43.280479 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.281320 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.282405 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:43.282854 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:43.283024 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:43.285949 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:43.285956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:43.285969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:43.286001 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:43.288067 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:43.288741 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:43.288747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:43.288753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:43.288902 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:43.289330 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:43.289334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:43.289335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:43.289790 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:43.289912 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:43.290531 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 14:59:43.290764 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
W20260619 14:59:43.293190 25458 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
W20260619 14:59:43.293352 25459 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
I20260619 14:59:43.293476 25459 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc; no voters: 6ffa6bed5f934e7cb4870a07766cb254, 8a00b83581344e6dbbe065a6ffb579db
I20260619 14:59:43.293720 25845 raft_consensus.cc:2726] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:59:43.959913 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:43.959563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:57461)
0619 14:59:43.959784 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:43.959828 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260619 14:59:44.091650 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:44.091326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:33867)
0619 14:59:44.091531 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:44.091578 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260619 14:59:44.268851 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:44.268533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:37457)
0619 14:59:44.268737 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:44.268780 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260619 14:59:44.961010 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:44.960632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:52099)
0619 14:59:44.960868 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:44.960918 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
I20260619 14:59:45.092625 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:45.092231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:37841)
0619 14:59:45.092492 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:45.092550 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260619 14:59:45.243737 25854 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:45.243850 25854 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:45.244268 25854 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
W20260619 14:59:45.245117 25458 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
W20260619 14:59:45.245214 25459 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
I20260619 14:59:45.245321 25459 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc; no voters: 6ffa6bed5f934e7cb4870a07766cb254, 8a00b83581344e6dbbe065a6ffb579db
I20260619 14:59:45.245448 25854 raft_consensus.cc:2726] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:59:45.269758 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:45.269428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:57265)
0619 14:59:45.269641 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:45.269685 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260619 14:59:45.962026 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:45.961661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:36109)
0619 14:59:45.961896 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:45.961941 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260619 14:59:46.093626 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:46.093222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:42979)
0619 14:59:46.093500 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:46.093550 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260619 14:59:46.270696 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:46.270357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:49705)
0619 14:59:46.270567 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:46.270614 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":59,"threads_started":1}
I20260619 14:59:46.963340 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:46.962888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:54327)
0619 14:59:46.963195 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:46.963247 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260619 14:59:47.094632 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:47.094287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:37283)
0619 14:59:47.094501 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:47.094552 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260619 14:59:47.271708 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:47.271357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:49667)
0619 14:59:47.271572 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:47.271631 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260619 14:59:47.439245 25862 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:47.439390 25862 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:47.439821 25862 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
W20260619 14:59:47.440604 25458 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
W20260619 14:59:47.440711 25459 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
I20260619 14:59:47.440828 25459 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc; no voters: 6ffa6bed5f934e7cb4870a07766cb254, 8a00b83581344e6dbbe065a6ffb579db
I20260619 14:59:47.441028 25862 raft_consensus.cc:2726] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:59:47.964459 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:47.964018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:46803)
0619 14:59:47.964286 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:47.964366 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260619 14:59:48.095633 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:48.095271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:38851)
0619 14:59:48.095490 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:48.095541 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260619 14:59:48.272687 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:48.272276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:56737)
0619 14:59:48.272561 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:48.272610 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":68,"threads_started":1}
I20260619 14:59:48.965605 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:48.965201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:33207)
0619 14:59:48.965459 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:48.965511 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260619 14:59:49.096752 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:49.096349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:39213)
0619 14:59:49.096620 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:49.096671 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":70,"threads_started":1}
I20260619 14:59:49.273898 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:49.273356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:50941)
0619 14:59:49.273622 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:49.273673 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":64,"threads_started":1}
I20260619 14:59:49.966646 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:49.966280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:55339)
0619 14:59:49.966505 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:49.966556 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260619 14:59:50.097721 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:50.097392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:38649)
0619 14:59:50.097599 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:50.097646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260619 14:59:50.274974 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:50.274529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:57395)
0619 14:59:50.274771 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:50.274852 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260619 14:59:50.735564 25872 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:50.735818 25872 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:50.736442 25872 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
W20260619 14:59:50.737104 25459 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
W20260619 14:59:50.737272 25458 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
I20260619 14:59:50.737337 25458 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc; no voters: 6ffa6bed5f934e7cb4870a07766cb254, 8a00b83581344e6dbbe065a6ffb579db
I20260619 14:59:50.737466 25872 raft_consensus.cc:2726] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:59:50.967805 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:50.967360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:46557)
0619 14:59:50.967632 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:50.967682 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":69,"threads_started":1}
I20260619 14:59:51.098639 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:51.098313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:38977)
0619 14:59:51.098513 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:51.098564 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260619 14:59:51.276036 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:51.275649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:55651)
0619 14:59:51.275905 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:51.275955 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":59,"threads_started":1}
I20260619 14:59:51.969017 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:51.968645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:60797)
0619 14:59:51.968871 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:51.968919 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260619 14:59:52.099622 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:52.099279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:43351)
0619 14:59:52.099490 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:52.099541 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260619 14:59:52.277078 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:52.276754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:51871)
0619 14:59:52.276953 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:52.277004 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260619 14:59:52.970126 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:52.969740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:58911)
0619 14:59:52.969971 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:52.970038 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260619 14:59:53.100829 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:53.100400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:35299)
0619 14:59:53.100665 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:53.100726 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":63,"threads_started":1}
I20260619 14:59:53.278065 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:53.277711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:34139)
0619 14:59:53.277932 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:53.277981 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260619 14:59:53.971494 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:53.971016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:41623)
0619 14:59:53.971332 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:53.971399 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":114,"threads_started":1}
I20260619 14:59:54.101891 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:54.101508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:60059)
0619 14:59:54.101744 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:54.101809 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260619 14:59:54.279024 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:54.278661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:54231)
0619 14:59:54.278883 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:54.278933 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260619 14:59:54.972671 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:54.972244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:52877)
0619 14:59:54.972531 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:54.972582 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":117,"threads_started":1}
I20260619 14:59:55.102831 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:55.102499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:39931)
0619 14:59:55.102711 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:55.102756 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260619 14:59:55.280012 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:55.279637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:56907)
0619 14:59:55.279863 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:55.279907 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260619 14:59:55.664441 25888 raft_consensus.cc:482] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:55.664570 25888 raft_consensus.cc:504] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ffa6bed5f934e7cb4870a07766cb254" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33199 } } peers { permanent_uuid: "8a00b83581344e6dbbe065a6ffb579db" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 36035 } } peers { permanent_uuid: "3dbef50bf0444769b9bd4e2eda487dfc" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 45727 } }
I20260619 14:59:55.664955 25888 leader_election.cc:288] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199), 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035)
W20260619 14:59:55.665650 25458 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a00b83581344e6dbbe065a6ffb579db (127.10.71.131:36035): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
W20260619 14:59:55.665802 25459 leader_election.cc:341] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ffa6bed5f934e7cb4870a07766cb254 (127.10.71.130:33199): Not found: Tablet not found: 97dbbb978aef42cc9109c952917f6743
I20260619 14:59:55.665874 25459 leader_election.cc:302] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dbef50bf0444769b9bd4e2eda487dfc; no voters: 6ffa6bed5f934e7cb4870a07766cb254, 8a00b83581344e6dbbe065a6ffb579db
I20260619 14:59:55.666054 25888 raft_consensus.cc:2726] T 97dbbb978aef42cc9109c952917f6743 P 3dbef50bf0444769b9bd4e2eda487dfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:59:55.973701 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:55.973315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:39893)
0619 14:59:55.973565 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:55.973614 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260619 14:59:56.103739 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:56.103389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:38885)
0619 14:59:56.103609 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:56.103662 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260619 14:59:56.280921 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:56.280594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:49311)
0619 14:59:56.280799 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:56.280845 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260619 14:59:56.974938 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:56.974548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.129:37861)
0619 14:59:56.974802 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:56.974851 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260619 14:59:57.104836 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:57.104449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.130:50049)
0619 14:59:57.104682 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:57.104755 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":63,"threads_started":1}
I20260619 14:59:57.271843 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 25407
I20260619 14:59:57.277418 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 25574
I20260619 14:59:57.281944 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:57.281622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42951 (local address 127.10.71.131:60251)
0619 14:59:57.281831 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:57.281874 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260619 14:59:57.282517 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 25709
2026-06-19T14:59:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19041 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:59:57 dist-test-slave-3qdh krb5kdc[25897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:59:57 dist-test-slave-3qdh krb5kdc[25897](info): set up 2 sockets
Jun 19 14:59:57 dist-test-slave-3qdh krb5kdc[25897](info): commencing operation
krb5kdc: starting...
W20260619 14:59:58.236194 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:59:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 14:59:58.340963 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38943
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:58.439363 25910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:58.439551 25910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:58.439586 25910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:58.442879 25910 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:59:58.442975 25910 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:59:58.442994 25910 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:59:58.443009 25910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:59:58.443028 25910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:58.443048 25910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:58.443063 25910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:59:58.443079 25910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:59:58.447634 25910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:58.448799 25910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:58.449545 25910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:58.457028 25915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:58.457091 25910 server_base.cc:1034] running on GCE node
W20260619 14:59:58.457020 25916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:58.457039 25918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:58.457610 25910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:58.458432 25910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:58.459601 25910 hybrid_clock.cc:648] HybridClock initialized: now 1781881198459588 us; error 28 us; skew 500 ppm
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:58.461871 25910 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 14:59:58.462558 25910 webserver.cc:460] Webserver started at http://127.10.71.190:39127/ using document root <none> and password file <none>
I20260619 14:59:58.462980 25910 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:58.463063 25910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:58.463241 25910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:58.464898 25910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "85a80d08786b49a4a2d0b1ae027407f3"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "1bab2407a41a0c9cb3157d986f5ac344"
server_key_iv: "d45c5dd3386641ed89b6263be3635260"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.465378 25910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "85a80d08786b49a4a2d0b1ae027407f3"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "1bab2407a41a0c9cb3157d986f5ac344"
server_key_iv: "d45c5dd3386641ed89b6263be3635260"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.468472 25910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:59:58.470391 25925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.471371 25910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:58.471493 25910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "85a80d08786b49a4a2d0b1ae027407f3"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "1bab2407a41a0c9cb3157d986f5ac344"
server_key_iv: "d45c5dd3386641ed89b6263be3635260"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.471583 25910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:58.480105 25910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:58.482224 25910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:58.482390 25910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:58.488808 25910 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:38943
I20260619 14:59:58.489049 25977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:38943 every 8 connection(s)
I20260619 14:59:58.489612 25910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 14:59:58.491761 25978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:58.495325 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 25910
I20260619 14:59:58.495434 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 14:59:58.495743 10526 external_mini_cluster.cc:1428] Setting key 31810e2d8e3026b6993f57b24570e96e
I20260619 14:59:58.496289 25978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap starting.
I20260619 14:59:58.498251 25978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:58.498857 25978 log.cc:824] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:58.500622 25978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: No bootstrap required, opened a new log
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:58.504025 25978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 14:59:58.504245 25978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:58.504290 25978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a80d08786b49a4a2d0b1ae027407f3, State: Initialized, Role: FOLLOWER
I20260619 14:59:58.504765 25978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 14:59:58.504940 25978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:59:58.505004 25978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:59:58.505095 25978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:58.506000 25978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 14:59:58.506304 25978 leader_election.cc:302] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a80d08786b49a4a2d0b1ae027407f3; no voters:
I20260619 14:59:58.506605 25978 leader_election.cc:288] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:59:58.506669 25983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:58.506868 25983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 LEADER]: Becoming Leader. State: Replica: 85a80d08786b49a4a2d0b1ae027407f3, State: Running, Role: LEADER
I20260619 14:59:58.507174 25983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 14:59:58.507687 25978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:59:58.508847 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:58.496944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.10.71.190:38943)
0619 14:59:58.497289 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:58.497300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:58.497327 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:59:58.497822 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:58.497834 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:58.498084 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:58.498359 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:58.499065 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.500284 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.501078 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.501627 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.501802 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:58.503965 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:58.503970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:58.503981 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:58.504015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:58.505774 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:58.506197 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:58.506202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:58.506207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:58.506320 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:58.506596 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:58.506600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:58.506602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:58.507084 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:58.507279 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:58.507835 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 14:59:58.508052 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260619 14:59:58.510041 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a80d08786b49a4a2d0b1ae027407f3. Latest consensus state: current_term: 1 leader_uuid: "85a80d08786b49a4a2d0b1ae027407f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } } }
I20260619 14:59:58.510190 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:59:58.510203 25984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a80d08786b49a4a2d0b1ae027407f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } } }
I20260619 14:59:58.510273 25984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: This master's current role is: LEADER
W20260619 14:59:58.512025 25998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:59:58.512079 25998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:59:58.512167 25999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:59:58.512823 25999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:59:58.517302 25999 catalog_manager.cc:1347] Generated new cluster ID: 5ec36fd2c1ad4744a914f30badffa6ef
I20260619 14:59:58.517354 25999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:59:58.534535 25999 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:59:58.535173 25999 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:59:58.541085 25999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Generated new TSK 0
I20260619 14:59:58.541676 25999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 14:59:58.567329 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38943
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:58.666055 26004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:58.666298 26004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:58.666412 26004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:58.670024 26004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:58.670094 26004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:58.670169 26004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 14:59:58.674582 26004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:58.675355 26004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:58.676208 26004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:58.684319 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:58.684330 26009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:58.684350 26004 server_base.cc:1034] running on GCE node
W20260619 14:59:58.684288 26010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:58.684885 26004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:58.685483 26004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:58.686693 26004 hybrid_clock.cc:648] HybridClock initialized: now 1781881198686664 us; error 41 us; skew 500 ppm
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:58.689110 26004 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 14:59:58.689761 26004 webserver.cc:460] Webserver started at http://127.10.71.129:46255/ using document root <none> and password file <none>
I20260619 14:59:58.690209 26004 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:58.690284 26004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:58.690469 26004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:58.692045 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "2ef20d88415141dfad7d62f547cffb7b"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "10d79e580e9ac03439977b0ebee4d774"
server_key_iv: "42bf510856a1b2f1b58566dfc7cfc16e"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.692553 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "2ef20d88415141dfad7d62f547cffb7b"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "10d79e580e9ac03439977b0ebee4d774"
server_key_iv: "42bf510856a1b2f1b58566dfc7cfc16e"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.695542 26004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:59:58.697646 26019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.698626 26004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:59:58.698755 26004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "2ef20d88415141dfad7d62f547cffb7b"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "10d79e580e9ac03439977b0ebee4d774"
server_key_iv: "42bf510856a1b2f1b58566dfc7cfc16e"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.698850 26004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:58.729823 26004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:58.732084 26004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:58.732247 26004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:58.732708 26004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:58.733436 26004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:58.733497 26004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.733557 26004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:58.733599 26004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.741807 26004 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:36951
I20260619 14:59:58.741832 26132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:36951 every 8 connection(s)
I20260619 14:59:58.742560 26004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:58.752509 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26004
I20260619 14:59:58.752627 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 14:59:58.752934 10526 external_mini_cluster.cc:1428] Setting key 3afdb47224b0ea1e13bd512494cefd5e
I20260619 14:59:58.754684 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:58.744138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:52297 (local address 127.10.71.190:38943)
0619 14:59:58.744265 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:58.744269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:58.745362 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 14:59:58.746111 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:58.746113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:58.746156 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:58.746229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:58.747241 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.747684 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.748784 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.749034 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.749123 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:58.751146 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:58.751149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:58.751151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:58.751172 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:58.752289 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:58.752875 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:58.752878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:58.752879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:58.752949 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:58.753332 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:58.753335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:58.753336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:58.753550 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:58.753638 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:58.754476 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 14:59:58.754578 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:59:58.755676 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:58.744515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:52297)
0619 14:59:58.745221 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:58.745258 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:58.745939 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:58.746370 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:58.746376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:58.746698 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:58.747119 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:58.747130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.747787 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.747789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:58.748662 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:58.748671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.749154 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.749156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:58.749272 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:58.749695 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:58.749708 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:58.750987 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:58.752446 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:58.752452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:58.752464 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:58.752789 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:58.753094 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:58.753096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:58.753098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:58.753240 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:58.754011 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:58.754018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:58.754258 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 14:59:58.754433 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 14:59:58.754729 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":427,"threads_started":1}
I20260619 14:59:58.756863 26133 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38943
I20260619 14:59:58.757102 26133 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:58.757552 26133 heartbeater.cc:511] Master 127.10.71.190:38943 requested a full tablet report, sending...
I20260619 14:59:58.758846 25940 ts_manager.cc:194] Registered new tserver with Master: 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951)
I20260619 14:59:58.759773 25940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:52297
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 14:59:58.776526 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38943
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:58.876618 26138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:58.876838 26138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:58.876911 26138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:58.880766 26138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:58.880821 26138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:58.880893 26138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 14:59:58.885535 26138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:58.886313 26138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:58.887137 26138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:58.895222 26146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:58.895308 26138 server_base.cc:1034] running on GCE node
W20260619 14:59:58.895234 26143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:58.895278 26144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:58.895763 26138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:58.896401 26138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:58.897572 26138 hybrid_clock.cc:648] HybridClock initialized: now 1781881198897566 us; error 27 us; skew 500 ppm
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:58.899806 26138 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 14:59:58.900626 26138 webserver.cc:460] Webserver started at http://127.10.71.130:39333/ using document root <none> and password file <none>
I20260619 14:59:58.901134 26138 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:58.901219 26138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:58.901420 26138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:58.903178 26138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "f5594b20150644e3b4ed8f8295c5ebbf"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "b3a276e74809f39168a7526dfb7c21c4"
server_key_iv: "4427591a71d710bb58a44777fa655678"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.903649 26138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "f5594b20150644e3b4ed8f8295c5ebbf"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "b3a276e74809f39168a7526dfb7c21c4"
server_key_iv: "4427591a71d710bb58a44777fa655678"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.906872 26138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:59:58.909050 26153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.910039 26138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:58.910157 26138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "f5594b20150644e3b4ed8f8295c5ebbf"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "b3a276e74809f39168a7526dfb7c21c4"
server_key_iv: "4427591a71d710bb58a44777fa655678"
server_key_version: "encryptionkey@0"
I20260619 14:59:58.910252 26138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:58.919992 26138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:58.922323 26138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:58.922498 26138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:58.922947 26138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:58.923600 26138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:58.923662 26138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.923720 26138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:58.923750 26138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:58.932065 26138 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:37541
I20260619 14:59:58.932082 26266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:37541 every 8 connection(s)
I20260619 14:59:58.932729 26138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:59:58 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881198, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:58.941007 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26138
I20260619 14:59:58.941149 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 14:59:58.941422 10526 external_mini_cluster.cc:1428] Setting key 99885ccd6223d9bb428d7847d1560bee
I20260619 14:59:58.944293 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:58.934090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:36133 (local address 127.10.71.190:38943)
0619 14:59:58.934236 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:58.934240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:58.935011 (+ 771us) server_negotiation.cc:407] Connection header received
0619 14:59:58.935714 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:58.935716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:58.935760 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:58.935830 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:58.936896 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.937433 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.938431 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.938631 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.938704 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:58.940766 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:58.940769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:58.940784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:58.940807 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:58.942353 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:58.942857 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:58.942859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:58.942861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:58.942941 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:58.943288 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:58.943290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:58.943291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:58.943433 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:58.943510 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:58.944022 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 14:59:58.944108 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:59:58.945006 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:58.934372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:36133)
0619 14:59:58.934885 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:58.934916 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:58.935540 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:58.935973 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:58.935978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:58.936341 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:58.936762 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:58.936774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.937540 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.937543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:58.938328 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:58.938335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:58.938743 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:58.938746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:58.938859 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:58.939268 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:58.939281 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:58.940604 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:58.942466 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:58.942471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:58.942488 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:58.942762 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:58.943026 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:58.943032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:58.943035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:58.943205 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:58.943525 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:58.943533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:58.943791 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 14:59:58.943966 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 14:59:58.944265 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":102,"threads_started":1}
I20260619 14:59:58.946023 26267 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38943
I20260619 14:59:58.946233 26267 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:58.946643 26267 heartbeater.cc:511] Master 127.10.71.190:38943 requested a full tablet report, sending...
I20260619 14:59:58.947573 25940 ts_manager.cc:194] Registered new tserver with Master: f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
I20260619 14:59:58.948105 25940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:36133
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 14:59:58.965306 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38943
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:59:59.063285 26272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:59:59.063493 26272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:59:59.063531 26272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:59:59.067063 26272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:59:59.067111 26272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:59:59.067158 26272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 14:59:59.071395 26272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:59:59.072098 26272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:59:59.072894 26272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:59:59.081343 26278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:59:59.081347 26277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:59.081489 26272 server_base.cc:1034] running on GCE node
W20260619 14:59:59.081507 26280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:59:59.082089 26272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:59:59.082607 26272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:59:59.083775 26272 hybrid_clock.cc:648] HybridClock initialized: now 1781881199083768 us; error 24 us; skew 500 ppm
Jun 19 14:59:59 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881199, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:59:59.086083 26272 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 14:59:59.086802 26272 webserver.cc:460] Webserver started at http://127.10.71.131:42651/ using document root <none> and password file <none>
I20260619 14:59:59.087253 26272 fs_manager.cc:359] Metadata directory not provided
I20260619 14:59:59.087329 26272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:59:59.087513 26272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:59:59.089113 26272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "c22b25bed87141faac0d89c52e4e2d95"
format_stamp: "Formatted at 2026-06-19 14:59:59 on dist-test-slave-3qdh"
server_key: "ab4b6aa67488be2a52df3ea0e8a97757"
server_key_iv: "5f3a64cfdbcd2ade2f085aeb03ffa5b8"
server_key_version: "encryptionkey@0"
I20260619 14:59:59.089569 26272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "c22b25bed87141faac0d89c52e4e2d95"
format_stamp: "Formatted at 2026-06-19 14:59:59 on dist-test-slave-3qdh"
server_key: "ab4b6aa67488be2a52df3ea0e8a97757"
server_key_iv: "5f3a64cfdbcd2ade2f085aeb03ffa5b8"
server_key_version: "encryptionkey@0"
I20260619 14:59:59.092558 26272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:59:59.094396 26287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:59.095398 26272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:59:59.095543 26272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "c22b25bed87141faac0d89c52e4e2d95"
format_stamp: "Formatted at 2026-06-19 14:59:59 on dist-test-slave-3qdh"
server_key: "ab4b6aa67488be2a52df3ea0e8a97757"
server_key_iv: "5f3a64cfdbcd2ade2f085aeb03ffa5b8"
server_key_version: "encryptionkey@0"
I20260619 14:59:59.095674 26272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:59:59.115424 26272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:59:59.117756 26272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:59:59.117918 26272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:59:59.118348 26272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:59:59.118950 26272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:59:59.119016 26272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:59.119074 26272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:59:59.119103 26272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:59:59.126199 26272 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:40341
I20260619 14:59:59.126226 26400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:40341 every 8 connection(s)
I20260619 14:59:59.126917 26272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 14:59:59.130043 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26272
I20260619 14:59:59.130173 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 14:59:59.130470 10526 external_mini_cluster.cc:1428] Setting key 8161408c5ea2940078f5148ac2835d7d
Jun 19 14:59:59 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881199, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 14:59:59.138163 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.128368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:43987 (local address 127.10.71.190:38943)
0619 14:59:59.128489 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.128492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.129244 (+ 752us) server_negotiation.cc:407] Connection header received
0619 14:59:59.130005 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.130009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.130054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.130148 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:59.131319 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.131859 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.132863 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.133076 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.133177 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.135105 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:59.135108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:59.135110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:59.135130 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:59.136395 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:59.136862 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:59.136867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:59.136869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:59.136958 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:59.137276 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:59.137281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:59.137282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:59.137446 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:59.137538 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.137918 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.138026 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:59:59.138700 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.128613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:43987)
0619 14:59:59.129128 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.129159 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:59.129763 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:59.130296 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:59.130304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:59.130741 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:59:59.131183 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.131199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.131969 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.131972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.132750 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.132758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.133201 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.133205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.133358 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.133737 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:59:59.133749 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:59:59.134945 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:59:59.136502 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:59.136507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:59.136527 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:59.136759 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:59.137051 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:59:59.137054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:59:59.137056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:59:59.137189 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:59:59.137542 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:59:59.137548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:59:59.137730 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 14:59:59.137882 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:59:59.138116 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":95,"threads_started":1}
I20260619 14:59:59.139617 26401 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38943
I20260619 14:59:59.139796 26401 heartbeater.cc:464] Registering TS with master...
I20260619 14:59:59.140158 26401 heartbeater.cc:511] Master 127.10.71.190:38943 requested a full tablet report, sending...
I20260619 14:59:59.141062 25940 ts_manager.cc:194] Registered new tserver with Master: c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341)
I20260619 14:59:59.141496 25940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:43987
I20260619 14:59:59.143786 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:59:59.152004 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.145574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.10.71.190:38943)
0619 14:59:59.145745 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.145751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.145952 (+ 201us) server_negotiation.cc:407] Connection header received
0619 14:59:59.146081 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.146084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.146121 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.146210 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:59:59.146860 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.147321 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.148076 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.148279 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.148397 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.149481 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:59:59.149485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:59:59.149487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:59:59.149509 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:59:59.150543 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:59.150909 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:59.150913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:59.150914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:59.150982 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:59:59.151259 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:59:59.151263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:59:59.151265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:59:59.151439 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:59:59.151568 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.151772 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.151866 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:59:59.154253 25940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:59:59.156932 25940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:59:59.169576 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.163925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:40341 (local address 127.0.0.1:47242)
0619 14:59:59.164549 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.164559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:59.164654 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:59.165319 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:59.165324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:59.165346 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:59.165463 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.165480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.166902 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.166905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.167995 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.168005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.169090 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.169094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.169193 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.169230 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:59:59.169327 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:59:59.169428 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":7,"thread_start_us":89,"threads_started":1}
I20260619 14:59:59.169620 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.164285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37541 (local address 127.0.0.1:60426)
0619 14:59:59.164963 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.164975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:59.165065 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:59.165835 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:59.165837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:59.165851 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:59.165918 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.165922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.167107 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.167111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.168190 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.168200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.169285 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.169291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.169377 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.169395 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:59:59.169463 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:59:59.169523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260619 14:59:59.169765 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.164459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.10.71.130:37541)
0619 14:59:59.164850 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.164860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.165148 (+ 288us) server_negotiation.cc:407] Connection header received
0619 14:59:59.165228 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.165235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.165485 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.165639 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:59.166052 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.166963 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.168379 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.169138 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.169248 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.169373 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.169520 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.169605 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":126,"threads_started":1}
I20260619 14:59:59.170002 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.164435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.10.71.131:40341)
0619 14:59:59.164754 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.164764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.164794 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:59:59.164960 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.164966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.165143 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.165275 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:59.165621 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.166740 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.168126 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.168964 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.169088 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.169208 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.169846 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.169917 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":138,"threads_started":1}
I20260619 14:59:59.170567 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.163703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36951 (local address 127.0.0.1:35290)
0619 14:59:59.164167 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.164209 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:59.164337 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:59.166155 (+ 1818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:59.166157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:59.166177 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:59.166256 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.166262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.167588 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.167591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.168708 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.168719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.170281 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.170283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.170342 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.170357 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 14:59:59.170396 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 14:59:59.170438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":62,"threads_started":1}
I20260619 14:59:59.171182 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.164817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.10.71.129:36951)
0619 14:59:59.165591 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.165604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.165631 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:59:59.165743 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.165749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.165931 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.166153 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:59.166377 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.167450 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.168841 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.169986 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.170805 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.170930 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.170972 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.171059 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":143,"threads_started":1}
I20260619 14:59:59.172338 26201 tablet_service.cc:1467] Processing CreateTablet for tablet d270d32f7ed341748208a97e0eb83cf4 (DEFAULT_TABLE table=test-table [id=d7b400d4e5b1405cbaa57b379180a825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:59.172343 26335 tablet_service.cc:1467] Processing CreateTablet for tablet d270d32f7ed341748208a97e0eb83cf4 (DEFAULT_TABLE table=test-table [id=d7b400d4e5b1405cbaa57b379180a825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:59.173174 26201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d270d32f7ed341748208a97e0eb83cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:59.173249 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d270d32f7ed341748208a97e0eb83cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:59.173404 26067 tablet_service.cc:1467] Processing CreateTablet for tablet d270d32f7ed341748208a97e0eb83cf4 (DEFAULT_TABLE table=test-table [id=d7b400d4e5b1405cbaa57b379180a825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:59:59.174227 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d270d32f7ed341748208a97e0eb83cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:59:59.178699 26418 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Bootstrap starting.
I20260619 14:59:59.180089 26419 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Bootstrap starting.
I20260619 14:59:59.180969 26420 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap starting.
I20260619 14:59:59.181404 26418 tablet_bootstrap.cc:654] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:59.182075 26419 tablet_bootstrap.cc:654] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:59.182233 26418 log.cc:824] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:59.183185 26420 tablet_bootstrap.cc:654] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Neither blocks nor log segments found. Creating new log.
I20260619 14:59:59.183282 26419 log.cc:824] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:59.183914 26420 log.cc:824] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:59:59.184057 26418 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: No bootstrap required, opened a new log
I20260619 14:59:59.184237 26418 ts_tablet_manager.cc:1397] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:59:59.185024 26419 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: No bootstrap required, opened a new log
I20260619 14:59:59.185189 26419 ts_tablet_manager.cc:1397] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:59:59.185441 26420 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: No bootstrap required, opened a new log
I20260619 14:59:59.185616 26420 ts_tablet_manager.cc:1397] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:59:59.187954 26418 raft_consensus.cc:348] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.188211 26418 raft_consensus.cc:374] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:59.188401 26418 raft_consensus.cc:729] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5594b20150644e3b4ed8f8295c5ebbf, State: Initialized, Role: FOLLOWER
I20260619 14:59:59.188496 26420 raft_consensus.cc:348] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.188723 26420 raft_consensus.cc:374] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:59.188783 26420 raft_consensus.cc:729] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef20d88415141dfad7d62f547cffb7b, State: Initialized, Role: FOLLOWER
I20260619 14:59:59.188865 26418 consensus_queue.cc:260] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.188817 26419 raft_consensus.cc:348] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.189069 26419 raft_consensus.cc:374] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:59:59.189132 26419 raft_consensus.cc:729] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22b25bed87141faac0d89c52e4e2d95, State: Initialized, Role: FOLLOWER
I20260619 14:59:59.189158 26420 consensus_queue.cc:260] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.189550 26267 heartbeater.cc:503] Master 127.10.71.190:38943 was elected leader, sending a full tablet report...
I20260619 14:59:59.189536 26419 consensus_queue.cc:260] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.189908 26418 ts_tablet_manager.cc:1428] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:59:59.190044 26420 ts_tablet_manager.cc:1428] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:59:59.190380 26133 heartbeater.cc:503] Master 127.10.71.190:38943 was elected leader, sending a full tablet report...
I20260619 14:59:59.190426 26419 ts_tablet_manager.cc:1428] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 14:59:59.190673 26401 heartbeater.cc:503] Master 127.10.71.190:38943 was elected leader, sending a full tablet report...
W20260619 14:59:59.243534 26134 tablet.cc:2369] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:59:59.297020 26425 raft_consensus.cc:482] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:59:59.297155 26425 raft_consensus.cc:504] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.298086 26425 leader_election.cc:288] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
I20260619 14:59:59.301689 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.298226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:40341 (local address 127.10.71.129:55285)
0619 14:59:59.298513 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.298528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:59.298648 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:59.298904 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:59.298907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:59.298923 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:59.299040 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.299047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.299710 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.299712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.300728 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.300737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.301352 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.301355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.301415 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.301433 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:59:59.301484 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 14:59:59.301582 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"spinlock_wait_cycles":8704,"thread_start_us":72,"threads_started":1}
I20260619 14:59:59.301690 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.298292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:55285 (local address 127.10.71.131:40341)
0619 14:59:59.298409 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.298413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.298601 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:59:59.298763 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.298768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.298814 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.298891 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:59.299140 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.299592 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.300836 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.301236 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.301366 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.301421 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.301518 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.301580 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:59:59.302279 26355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d270d32f7ed341748208a97e0eb83cf4" candidate_uuid: "2ef20d88415141dfad7d62f547cffb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22b25bed87141faac0d89c52e4e2d95" is_pre_election: true
I20260619 14:59:59.302366 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.298290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37541 (local address 127.10.71.129:37309)
0619 14:59:59.298516 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.298528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:59:59.298648 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:59:59.299462 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:59:59.299464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:59:59.299478 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:59:59.299569 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.299572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.300324 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.300328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.301193 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:59:59.301200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.301982 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.301986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:59:59.302080 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.302106 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:59:59.302201 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:59:59.302250 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"spinlock_wait_cycles":19584,"thread_start_us":65,"threads_started":1}
I20260619 14:59:59.302392 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.298382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:37309 (local address 127.10.71.130:37541)
0619 14:59:59.298540 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.298543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.298624 (+ 81us) server_negotiation.cc:407] Connection header received
0619 14:59:59.299004 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.299008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.299053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.299120 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:59:59.299674 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.300207 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.301309 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.301869 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.301998 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.302043 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.302223 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.302293 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:59:59.302551 26355 raft_consensus.cc:2454] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef20d88415141dfad7d62f547cffb7b in term 0.
I20260619 14:59:59.302944 26020 leader_election.cc:302] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b, c22b25bed87141faac0d89c52e4e2d95; no voters:
I20260619 14:59:59.303074 26221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d270d32f7ed341748208a97e0eb83cf4" candidate_uuid: "2ef20d88415141dfad7d62f547cffb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" is_pre_election: true
I20260619 14:59:59.303162 26425 raft_consensus.cc:2781] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:59:59.303236 26425 raft_consensus.cc:482] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:59:59.303280 26425 raft_consensus.cc:3037] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:59.303313 26221 raft_consensus.cc:2454] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef20d88415141dfad7d62f547cffb7b in term 0.
I20260619 14:59:59.304191 26425 raft_consensus.cc:504] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.304553 26425 leader_election.cc:288] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 1 election: Requested vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
I20260619 14:59:59.304805 26355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d270d32f7ed341748208a97e0eb83cf4" candidate_uuid: "2ef20d88415141dfad7d62f547cffb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22b25bed87141faac0d89c52e4e2d95"
I20260619 14:59:59.304906 26355 raft_consensus.cc:3037] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:59.304946 26221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d270d32f7ed341748208a97e0eb83cf4" candidate_uuid: "2ef20d88415141dfad7d62f547cffb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5594b20150644e3b4ed8f8295c5ebbf"
I20260619 14:59:59.305049 26221 raft_consensus.cc:3037] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:59:59.305866 26355 raft_consensus.cc:2454] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef20d88415141dfad7d62f547cffb7b in term 1.
I20260619 14:59:59.306007 26221 raft_consensus.cc:2454] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef20d88415141dfad7d62f547cffb7b in term 1.
I20260619 14:59:59.306162 26020 leader_election.cc:302] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b, c22b25bed87141faac0d89c52e4e2d95; no voters:
I20260619 14:59:59.306315 26425 raft_consensus.cc:2781] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:59:59.306497 26425 raft_consensus.cc:686] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 LEADER]: Becoming Leader. State: Replica: 2ef20d88415141dfad7d62f547cffb7b, State: Running, Role: LEADER
I20260619 14:59:59.306793 26425 consensus_queue.cc:237] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 14:59:59.309479 25942 catalog_manager.cc:5495] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129). New cstate: current_term: 1 leader_uuid: "2ef20d88415141dfad7d62f547cffb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260619 14:59:59.354640 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.351747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.10.71.129:36951)
0619 14:59:59.351875 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:59:59.351878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:59:59.351922 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:59:59.352064 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:59:59.352066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:59:59.352109 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:59:59.352181 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:59:59.352399 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.352799 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.353540 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:59:59.353720 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:59:59.353788 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:59:59.354066 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:59:59.354382 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:59:59.354457 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:59:59.354523 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:59:59.354560 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:59:59.359647 26355 raft_consensus.cc:1261] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Refusing update from remote peer 2ef20d88415141dfad7d62f547cffb7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:59.359648 26221 raft_consensus.cc:1261] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Refusing update from remote peer 2ef20d88415141dfad7d62f547cffb7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:59:59.360283 26429 consensus_queue.cc:1035] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:59.360472 26425 consensus_queue.cc:1035] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:59:59.366353 26432 mvcc.cc:205] Tried to move back new op lower bound from 7298585392570966016 to 7298585392362905600. Current Snapshot: MvccSnapshot[applied={T|T < 7298585392570966016}]
I20260619 14:59:59.369166 26434 mvcc.cc:205] Tried to move back new op lower bound from 7298585392570966016 to 7298585392362905600. Current Snapshot: MvccSnapshot[applied={T|T < 7298585392570966016}]
I20260619 14:59:59.373754 25938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:59.373885 25938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:59:59.375941 25938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Sending DeleteTablet for 3 replicas of tablet d270d32f7ed341748208a97e0eb83cf4
I20260619 14:59:59.376709 26067 tablet_service.cc:1514] Processing DeleteTablet for tablet d270d32f7ed341748208a97e0eb83cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:59 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:35290
I20260619 14:59:59.376828 26335 tablet_service.cc:1514] Processing DeleteTablet for tablet d270d32f7ed341748208a97e0eb83cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:59 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:47242
I20260619 14:59:59.376852 26201 tablet_service.cc:1514] Processing DeleteTablet for tablet d270d32f7ed341748208a97e0eb83cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:59 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:60426
I20260619 14:59:59.377169 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26004
I20260619 14:59:59.377226 26443 tablet_replica.cc:330] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: stopping tablet replica
I20260619 14:59:59.378055 26443 raft_consensus.cc:2229] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:59.378288 26443 raft_consensus.cc:2258] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:59.379297 26443 ts_tablet_manager.cc:1905] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:59.380645 26444 tablet_replica.cc:330] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: stopping tablet replica
I20260619 14:59:59.380908 26444 raft_consensus.cc:2229] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:59:59.381203 26444 raft_consensus.cc:2258] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:59:59.381449 26443 ts_tablet_manager.cc:1918] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:59:59.381570 26443 log.cc:1196] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/d270d32f7ed341748208a97e0eb83cf4
I20260619 14:59:59.381879 26443 ts_tablet_manager.cc:1939] T d270d32f7ed341748208a97e0eb83cf4 P c22b25bed87141faac0d89c52e4e2d95: Deleting consensus metadata
I20260619 14:59:59.382597 26444 ts_tablet_manager.cc:1905] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:59:59.382802 25926 catalog_manager.cc:4841] TS c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): tablet d270d32f7ed341748208a97e0eb83cf4 (table test-table [id=d7b400d4e5b1405cbaa57b379180a825]) successfully deleted
W20260619 14:59:59.383694 25927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:36951 (error 108)
I20260619 14:59:59.383852 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26138
I20260619 14:59:59.385743 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.385382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36951 (local address 127.0.0.1:35316)
0619 14:59:59.385557 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.385636 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:59:59.385906 25927 catalog_manager.cc:4584] TS 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d270d32f7ed341748208a97e0eb83cf4: Network error: Client connection negotiation failed: client connection to 127.10.71.129:36951: connect: Connection refused (error 111)
I20260619 14:59:59.389492 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26272
I20260619 14:59:59.390945 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:59:59.389541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37541 (local address 127.0.0.1:60438)
0619 14:59:59.390810 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0619 14:59:59.390870 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.130:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1175}
I20260619 14:59:59.394395 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 25910
I20260619 15:00:19.400243 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38943
--webserver_interface=127.10.71.190
--webserver_port=39127
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:19.497218 26446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:19.497434 26446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:19.497473 26446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:19.501091 26446 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 15:00:19.501130 26446 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 15:00:19.501144 26446 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 15:00:19.501158 26446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:00:19.501174 26446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:19.501220 26446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:19.501238 26446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:00:19.501268 26446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:00:19.505663 26446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=39127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:19.506438 26446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:19.507287 26446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:19.519052 26455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:19.519030 26453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:19.519083 26446 server_base.cc:1034] running on GCE node
W20260619 15:00:19.519022 26452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:19.519707 26446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:19.520721 26446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:19.521914 26446 hybrid_clock.cc:648] HybridClock initialized: now 1781881219521889 us; error 34 us; skew 500 ppm
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:19.528873 26446 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:00:19.529621 26446 webserver.cc:460] Webserver started at http://127.10.71.190:39127/ using document root <none> and password file <none>
I20260619 15:00:19.530110 26446 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:19.530202 26446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:19.533553 26446 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:00:19.535413 26462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:19.536530 26446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:19.536650 26446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "85a80d08786b49a4a2d0b1ae027407f3"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "1bab2407a41a0c9cb3157d986f5ac344"
server_key_iv: "d45c5dd3386641ed89b6263be3635260"
server_key_version: "encryptionkey@0"
I20260619 15:00:19.537164 26446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:19.570451 26446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:19.573228 26446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:19.573434 26446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:19.579955 26446 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:38943
I20260619 15:00:19.580202 26514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:38943 every 8 connection(s)
I20260619 15:00:19.580956 26446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:00:19.584858 26515 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:00:19.585638 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26446
I20260619 15:00:19.586032 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:36951
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:38943
--webserver_port=46255
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:00:19.586354 26515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap starting.
I20260619 15:00:19.607385 26515 log.cc:824] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:19.616591 26515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:00:19.616950 26515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap complete.
I20260619 15:00:19.621636 26515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:19.622038 26515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a80d08786b49a4a2d0b1ae027407f3, State: Initialized, Role: FOLLOWER
I20260619 15:00:19.622486 26515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:19.622615 26515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:00:19.622668 26515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:00:19.622745 26515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 15:00:19.624873 26515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:19.625175 26515 leader_election.cc:302] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a80d08786b49a4a2d0b1ae027407f3; no voters:
I20260619 15:00:19.625458 26515 leader_election.cc:288] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 15:00:19.625576 26519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 15:00:19.625897 26519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 LEADER]: Becoming Leader. State: Replica: 85a80d08786b49a4a2d0b1ae027407f3, State: Running, Role: LEADER
I20260619 15:00:19.626175 26519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:19.626474 26515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:00:19.627291 26520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85a80d08786b49a4a2d0b1ae027407f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } } }
I20260619 15:00:19.627403 26520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:00:19.628062 26521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a80d08786b49a4a2d0b1ae027407f3. Latest consensus state: current_term: 2 leader_uuid: "85a80d08786b49a4a2d0b1ae027407f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } } }
I20260619 15:00:19.628149 26521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:00:19.628931 26528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:00:19.632915 26528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:00:19.633496 26528 catalog_manager.cc:1259] Loaded cluster ID: 5ec36fd2c1ad4744a914f30badffa6ef
I20260619 15:00:19.633551 26528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:00:19.635766 26528 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:00:19.636381 26528 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Loaded TSK: 0
I20260619 15:00:19.648340 26528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Generated new TSK 1
I20260619 15:00:19.648543 26528 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:00:19.699461 26517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:19.699723 26517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:19.699800 26517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:19.703424 26517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:19.703490 26517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:19.703562 26517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:00:19.707985 26517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:36951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=46255
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:19.708913 26517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:19.709743 26517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:19.719028 26544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:19.719017 26543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:19.719077 26546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:19.719429 26517 server_base.cc:1034] running on GCE node
I20260619 15:00:19.719774 26517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:19.720330 26517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:19.721503 26517 hybrid_clock.cc:648] HybridClock initialized: now 1781881219721485 us; error 34 us; skew 500 ppm
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:19.724012 26517 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 15:00:19.724823 26517 webserver.cc:460] Webserver started at http://127.10.71.129:46255/ using document root <none> and password file <none>
I20260619 15:00:19.725323 26517 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:19.725389 26517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:19.729274 26517 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:00:19.731282 26553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:19.732447 26517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:00:19.732569 26517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "2ef20d88415141dfad7d62f547cffb7b"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "10d79e580e9ac03439977b0ebee4d774"
server_key_iv: "42bf510856a1b2f1b58566dfc7cfc16e"
server_key_version: "encryptionkey@0"
I20260619 15:00:19.733028 26517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:19.742313 26517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:19.745316 26517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:19.745493 26517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:19.745983 26517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:19.747012 26560 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 15:00:19.751582 26517 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 15:00:19.751647 26517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 15:00:19.751731 26517 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 15:00:19.753254 26517 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 15:00:19.753319 26517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 15:00:19.753577 26560 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap starting.
I20260619 15:00:19.763072 26517 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:36951
I20260619 15:00:19.763208 26667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:36951 every 8 connection(s)
I20260619 15:00:19.763916 26517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 15:00:19.772188 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26517
I20260619 15:00:19.772693 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:37541
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:38943
--webserver_port=39333
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:00:19.773674 26560 log.cc:824] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:00:19.779058 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.765637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:53301 (local address 127.10.71.190:38943)
0619 15:00:19.765987 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:19.766004 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:19.766421 (+ 417us) server_negotiation.cc:407] Connection header received
0619 15:00:19.767442 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:19.767452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:19.767766 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:19.768077 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:19.769160 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.769986 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.771127 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.771536 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.771709 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.774566 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:19.774577 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:19.774593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:19.774625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:19.776121 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.776624 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.776629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.776632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.776728 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.777103 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.777106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.777107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.777379 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:19.777565 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:19.778073 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 15:00:19.778316 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
I20260619 15:00:19.779070 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.765646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:53301)
0619 15:00:19.766277 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:19.766323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:19.767072 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:19.768126 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:19.768133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:19.768568 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:19.768984 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.768996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.770145 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.770148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.771002 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.771011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.771669 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.771672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.771842 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.772552 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:19.772582 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:19.774316 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:19.776231 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.776236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.776247 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.776512 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.776813 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.776817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.776819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.776963 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.777539 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:19.777545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:19.777776 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 15:00:19.777942 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 15:00:19.778244 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":147,"threads_started":1}
I20260619 15:00:19.781239 26668 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38943
I20260619 15:00:19.781442 26668 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:19.781883 26668 heartbeater.cc:511] Master 127.10.71.190:38943 requested a full tablet report, sending...
I20260619 15:00:19.781930 26560 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:00:19.782353 26560 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap complete.
I20260619 15:00:19.783622 26560 ts_tablet_manager.cc:1397] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260619 15:00:19.783882 26479 ts_manager.cc:194] Registered new tserver with Master: 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951)
I20260619 15:00:19.786135 26560 raft_consensus.cc:348] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:19.786645 26560 raft_consensus.cc:729] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef20d88415141dfad7d62f547cffb7b, State: Initialized, Role: FOLLOWER
I20260619 15:00:19.786784 26479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:53301
I20260619 15:00:19.787081 26560 consensus_queue.cc:260] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:19.788077 26560 ts_tablet_manager.cc:1428] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 15:00:19.789582 26668 heartbeater.cc:503] Master 127.10.71.190:38943 was elected leader, sending a full tablet report...
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.129@KRBTEST.COM
I20260619 15:00:19.796963 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.786185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36951 (local address 127.0.0.1:58714)
0619 15:00:19.786909 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:19.786945 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:19.787052 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:19.787827 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:19.787832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:19.788224 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:19.788363 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.788372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.789867 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.789872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.790831 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.790841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.791296 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.791299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.791390 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.791805 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:19.791817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:19.793360 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:19.795261 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.795269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.795273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.795555 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.795855 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.795858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.795860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.795963 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.796427 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:19.796433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:19.796549 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 15:00:19.796666 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 15:00:19.796823 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":74,"threads_started":1}
I20260619 15:00:19.797148 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.786955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.10.71.129:36951)
0619 15:00:19.787350 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:19.787360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:19.787379 (+ 19us) server_negotiation.cc:407] Connection header received
0619 15:00:19.787451 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:19.787456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:19.787643 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:19.788507 (+ 864us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:19.788862 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.789744 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.790927 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.791200 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.791271 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.793493 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:19.793515 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:19.793526 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:19.793546 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:19.795151 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.795660 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.795665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.795666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.795757 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.796054 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.796056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.796058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.796317 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:19.796419 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:19.796790 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 15:00:19.796929 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":73,"threads_started":1}
I20260619 15:00:19.798223 26602 tablet_service.cc:1514] Processing DeleteTablet for tablet d270d32f7ed341748208a97e0eb83cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:59 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:58714
I20260619 15:00:19.798960 26677 tablet_replica.cc:330] stopping tablet replica
I20260619 15:00:19.799197 26677 raft_consensus.cc:2229] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:00:19.799398 26677 raft_consensus.cc:2258] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:19.800192 26677 ts_tablet_manager.cc:1905] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:19.803503 26677 ts_tablet_manager.cc:1918] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:19.803611 26677 log.cc:1196] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal/wals/d270d32f7ed341748208a97e0eb83cf4
I20260619 15:00:19.803853 26677 ts_tablet_manager.cc:1939] T d270d32f7ed341748208a97e0eb83cf4 P 2ef20d88415141dfad7d62f547cffb7b: Deleting consensus metadata
I20260619 15:00:19.805333 26464 catalog_manager.cc:4841] TS 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951): tablet d270d32f7ed341748208a97e0eb83cf4 (table test-table [id=d7b400d4e5b1405cbaa57b379180a825]) successfully deleted
W20260619 15:00:19.878404 26672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:19.878616 26672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:19.878669 26672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:19.882329 26672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:19.882372 26672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:19.882421 26672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:00:19.886821 26672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:37541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=39333
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:19.887610 26672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:19.888504 26672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:19.896539 26685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:19.896596 26683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:19.896690 26672 server_base.cc:1034] running on GCE node
W20260619 15:00:19.896539 26682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:19.897194 26672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:19.897697 26672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:19.898844 26672 hybrid_clock.cc:648] HybridClock initialized: now 1781881219898822 us; error 37 us; skew 500 ppm
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:19.901286 26672 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 15:00:19.901937 26672 webserver.cc:460] Webserver started at http://127.10.71.130:39333/ using document root <none> and password file <none>
I20260619 15:00:19.902354 26672 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:19.902413 26672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:19.905447 26672 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:00:19.907204 26692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:19.908392 26672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:00:19.908478 26672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "f5594b20150644e3b4ed8f8295c5ebbf"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "b3a276e74809f39168a7526dfb7c21c4"
server_key_iv: "4427591a71d710bb58a44777fa655678"
server_key_version: "encryptionkey@0"
I20260619 15:00:19.908989 26672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:19.921959 26672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:19.924113 26672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:19.924233 26672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:19.924710 26672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:19.925576 26699 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 15:00:19.929229 26672 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 15:00:19.929302 26672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 15:00:19.929378 26672 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 15:00:19.930733 26672 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 15:00:19.930791 26672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 15:00:19.931018 26699 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Bootstrap starting.
I20260619 15:00:19.936527 26699 log.cc:824] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:19.940558 26672 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:37541
I20260619 15:00:19.940613 26807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:37541 every 8 connection(s)
I20260619 15:00:19.941386 26672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:00:19.943872 26699 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:00:19.944250 26699 tablet_bootstrap.cc:492] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Bootstrap complete.
I20260619 15:00:19.944564 26699 ts_tablet_manager.cc:1397] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20260619 15:00:19.946573 26699 raft_consensus.cc:348] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:19.946918 26699 raft_consensus.cc:729] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5594b20150644e3b4ed8f8295c5ebbf, State: Initialized, Role: FOLLOWER
I20260619 15:00:19.947353 26699 consensus_queue.cc:260] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:19.948175 26699 ts_tablet_manager.cc:1428] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 15:00:19.949476 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26672
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:00:19.949909 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:40341
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:38943
--webserver_port=42651
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:00:19.953852 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.942980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:59007 (local address 127.10.71.190:38943)
0619 15:00:19.943097 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:19.943101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:19.943629 (+ 528us) server_negotiation.cc:407] Connection header received
0619 15:00:19.944583 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:19.944587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:19.944645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:19.944718 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:19.945728 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.946183 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.947271 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.947546 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.947668 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.950302 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:19.950305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:19.950307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:19.950329 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:19.951888 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.952321 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.952327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.952331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.952408 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.952704 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.952709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.952714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.952959 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:19.953080 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:19.953463 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 15:00:19.953618 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 15:00:19.954324 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.943089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:59007)
0619 15:00:19.943504 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:19.943538 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:19.944388 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:19.944848 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:19.944854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:19.945198 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:19.945582 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.945593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.946333 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.946335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.947158 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.947167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.947675 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.947678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.947868 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.948679 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:19.948693 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:19.950158 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:19.951994 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.951998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.952006 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.952205 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.952495 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.952496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.952498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.952619 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.953063 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:19.953067 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:19.953247 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 15:00:19.953378 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 15:00:19.953646 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":140,"threads_started":1}
I20260619 15:00:19.955345 26808 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38943
I20260619 15:00:19.955559 26808 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:19.955909 26808 heartbeater.cc:511] Master 127.10.71.190:38943 requested a full tablet report, sending...
I20260619 15:00:19.957386 26479 ts_manager.cc:194] Registered new tserver with Master: f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
I20260619 15:00:19.958864 26479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:59007
I20260619 15:00:19.961308 26808 heartbeater.cc:503] Master 127.10.71.190:38943 was elected leader, sending a full tablet report...
Jun 19 15:00:19 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881219, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.130@KRBTEST.COM
I20260619 15:00:19.966351 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.958421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37541 (local address 127.0.0.1:34892)
0619 15:00:19.958824 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:19.958836 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:19.958929 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:19.959323 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:19.959330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:19.959531 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:19.959614 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.959619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.960627 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.960632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.961161 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:19.961168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.961803 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.961807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:19.961888 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.962303 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:19.962315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:19.963451 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:19.965053 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.965055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.965057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.965276 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.965503 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:19.965505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:19.965507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:19.965589 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:19.965919 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:19.965925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:19.966016 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 15:00:19.966114 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 15:00:19.966222 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337}
I20260619 15:00:19.966372 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:19.958494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.10.71.130:37541)
0619 15:00:19.958786 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:19.958795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:19.958902 (+ 107us) server_negotiation.cc:407] Connection header received
0619 15:00:19.959014 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:19.959018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:19.959203 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:19.959343 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:19.959719 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.960518 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.961265 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:19.961698 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:19.961802 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:19.963570 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:19.963576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:19.963584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:19.963601 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:19.964965 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.965366 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.965370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.965371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.965431 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:19.965662 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:19.965663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:19.965664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:19.965827 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:19.965909 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:19.966169 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 15:00:19.966267 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":218,"thread_start_us":139,"threads_started":1}
I20260619 15:00:19.967324 26741 tablet_service.cc:1514] Processing DeleteTablet for tablet d270d32f7ed341748208a97e0eb83cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:59:59 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34892
I20260619 15:00:19.968338 26814 tablet_replica.cc:330] stopping tablet replica
I20260619 15:00:19.968613 26814 raft_consensus.cc:2229] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:00:19.968832 26814 raft_consensus.cc:2258] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:19.969640 26814 ts_tablet_manager.cc:1905] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:19.972760 26814 ts_tablet_manager.cc:1918] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:19.972851 26814 log.cc:1196] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/d270d32f7ed341748208a97e0eb83cf4
I20260619 15:00:19.973078 26814 ts_tablet_manager.cc:1939] T d270d32f7ed341748208a97e0eb83cf4 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting consensus metadata
I20260619 15:00:19.974412 26464 catalog_manager.cc:4841] TS f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): tablet d270d32f7ed341748208a97e0eb83cf4 (table test-table [id=d7b400d4e5b1405cbaa57b379180a825]) successfully deleted
W20260619 15:00:20.052102 26812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:20.052366 26812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:20.052407 26812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:20.055879 26812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:20.055920 26812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:20.055968 26812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:00:20.060586 26812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:40341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=42651
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:20.061319 26812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:20.062052 26812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:20.070334 26822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:20.070334 26820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:20.070355 26819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:20.070547 26812 server_base.cc:1034] running on GCE node
I20260619 15:00:20.071015 26812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:20.071563 26812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:20.072729 26812 hybrid_clock.cc:648] HybridClock initialized: now 1781881220072704 us; error 41 us; skew 500 ppm
Jun 19 15:00:20 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:20.075048 26812 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 15:00:20.075830 26812 webserver.cc:460] Webserver started at http://127.10.71.131:42651/ using document root <none> and password file <none>
I20260619 15:00:20.076361 26812 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:20.076465 26812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:20.079761 26812 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:00:20.081558 26829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:20.082584 26812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 15:00:20.082721 26812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "c22b25bed87141faac0d89c52e4e2d95"
format_stamp: "Formatted at 2026-06-19 14:59:59 on dist-test-slave-3qdh"
server_key: "ab4b6aa67488be2a52df3ea0e8a97757"
server_key_iv: "5f3a64cfdbcd2ade2f085aeb03ffa5b8"
server_key_version: "encryptionkey@0"
I20260619 15:00:20.083170 26812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:20.102721 26812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:20.105206 26812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:20.105373 26812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:20.105845 26812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:20.106523 26812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:20.106593 26812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:20.106655 26812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:20.106706 26812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:20.115247 26812 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:40341
I20260619 15:00:20.115278 26942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:40341 every 8 connection(s)
I20260619 15:00:20.116114 26812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
Jun 19 15:00:20 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:00:20.126013 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26812
I20260619 15:00:20.128925 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.117806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:54633 (local address 127.10.71.190:38943)
0619 15:00:20.117942 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.117947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.118755 (+ 808us) server_negotiation.cc:407] Connection header received
0619 15:00:20.119887 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.119889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.119932 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.119999 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:20.121119 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.121620 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.122568 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.122743 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.122804 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.124898 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:20.124901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:20.124902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:20.124923 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:20.126506 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:20.127068 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:20.127131 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:20.127133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:20.127228 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:20.127602 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:20.127608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:20.127610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:20.127866 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:20.127969 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.128682 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.128798 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 15:00:20.130023 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.118189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:54633)
0619 15:00:20.118599 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:20.118637 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:20.119724 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:20.120163 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:20.120168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:20.120579 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:20.121005 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.121017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.121710 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.121712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.122475 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.122482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.122845 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.122847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.122960 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.123449 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:20.123462 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:20.124763 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:20.126608 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:20.126613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:20.126624 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:20.126921 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:20.127319 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:20.127322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:20.127324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:20.127504 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:20.127990 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:20.127995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:20.128351 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 15:00:20.129087 (+ 736us) client_negotiation.cc:240] Negotiation successful
0619 15:00:20.129393 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
I20260619 15:00:20.131243 26943 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38943
I20260619 15:00:20.131474 26943 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:20.131999 26943 heartbeater.cc:511] Master 127.10.71.190:38943 requested a full tablet report, sending...
I20260619 15:00:20.132975 26479 ts_manager.cc:194] Registered new tserver with Master: c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341)
I20260619 15:00:20.133683 26479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:54633
I20260619 15:00:20.134490 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.126654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.10.71.190:38943)
0619 15:00:20.126882 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.126887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.127041 (+ 154us) server_negotiation.cc:407] Connection header received
0619 15:00:20.127164 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.127167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.127231 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.127328 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:20.128250 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.128887 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.129757 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.129973 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.130074 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.131449 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:20.131452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:20.131455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:20.131486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:20.132950 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:20.133363 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:20.133366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:20.133367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:20.133447 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:20.133704 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:20.133706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:20.133707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:20.133887 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:20.134068 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.134243 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.134346 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260619 15:00:20.135987 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:00:20.139420 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.136592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.10.71.190:38943)
0619 15:00:20.136731 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.136734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.136933 (+ 199us) server_negotiation.cc:407] Connection header received
0619 15:00:20.137026 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.137028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.137096 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.137168 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:00:20.137334 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.137812 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.138552 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.138762 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.138837 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.139049 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:00:20.139154 (+ 105us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 15:00:20.139317 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46698: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 15:00:20.139484 26947 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46698: token expired
W20260619 15:00:20.139791 26407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.71.190:38943)
I20260619 15:00:20.145994 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.140101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.10.71.190:38943)
0619 15:00:20.140190 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.140193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.140243 (+ 50us) server_negotiation.cc:407] Connection header received
0619 15:00:20.140385 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.140388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.140425 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.140479 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:20.140958 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.141497 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.142203 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.142390 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.142470 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.143489 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:20.143492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:20.143493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:20.143511 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:20.144853 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:20.145176 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:20.145178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:20.145179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:20.145240 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:20.145475 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:20.145476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:20.145477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:20.145600 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:20.145682 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.145847 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.145914 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 15:00:20.147969 26479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:00:20.151011 26479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:00:20.157750 26741 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 (DEFAULT_TABLE table=test-table [id=9f58b3bfdc554de69beb3d2834657a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:20.157750 26602 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 (DEFAULT_TABLE table=test-table [id=9f58b3bfdc554de69beb3d2834657a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:20.158331 26602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:20.158331 26741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:20.161019 26699 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Bootstrap starting.
I20260619 15:00:20.161325 26560 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap starting.
I20260619 15:00:20.161623 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.156840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:40341 (local address 127.0.0.1:38576)
0619 15:00:20.156936 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:20.156952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:20.157041 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:20.158013 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:20.158016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:20.158036 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:20.158117 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.158121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.159608 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.159614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.160403 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.160412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.161226 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.161229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.161309 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.161352 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 15:00:20.161403 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 15:00:20.161485 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 15:00:20.162021 26699 tablet_bootstrap.cc:654] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:20.162127 26560 tablet_bootstrap.cc:654] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:20.162220 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.157124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.10.71.131:40341)
0619 15:00:20.157596 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.157605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.157623 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:00:20.157680 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.157684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.157833 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.158238 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:20.158352 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.159500 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.160506 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.161100 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.161785 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.161933 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.161984 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.162079 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":154,"threads_started":1}
I20260619 15:00:20.163434 26560 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: No bootstrap required, opened a new log
I20260619 15:00:20.163530 26560 ts_tablet_manager.cc:1397] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:20.163908 26560 raft_consensus.cc:348] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.164016 26560 raft_consensus.cc:374] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:20.164062 26560 raft_consensus.cc:729] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef20d88415141dfad7d62f547cffb7b, State: Initialized, Role: FOLLOWER
I20260619 15:00:20.164180 26560 consensus_queue.cc:260] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.164448 26560 ts_tablet_manager.cc:1428] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 15:00:20.165122 26699 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: No bootstrap required, opened a new log
I20260619 15:00:20.165217 26699 ts_tablet_manager.cc:1397] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 15:00:20.165086 26877 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 (DEFAULT_TABLE table=test-table [id=9f58b3bfdc554de69beb3d2834657a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:20.165556 26699 raft_consensus.cc:348] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.165686 26699 raft_consensus.cc:374] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:20.165722 26699 raft_consensus.cc:729] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5594b20150644e3b4ed8f8295c5ebbf, State: Initialized, Role: FOLLOWER
I20260619 15:00:20.165836 26699 consensus_queue.cc:260] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.166075 26877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:20.166079 26699 ts_tablet_manager.cc:1428] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 15:00:20.172549 26955 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Bootstrap starting.
I20260619 15:00:20.174479 26955 tablet_bootstrap.cc:654] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:20.175390 26955 log.cc:824] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:20.177102 26955 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: No bootstrap required, opened a new log
I20260619 15:00:20.177282 26955 ts_tablet_manager.cc:1397] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:00:20.179988 26955 raft_consensus.cc:348] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.180228 26955 raft_consensus.cc:374] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:20.180289 26955 raft_consensus.cc:729] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22b25bed87141faac0d89c52e4e2d95, State: Initialized, Role: FOLLOWER
I20260619 15:00:20.180749 26955 consensus_queue.cc:260] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.181365 26943 heartbeater.cc:503] Master 127.10.71.190:38943 was elected leader, sending a full tablet report...
I20260619 15:00:20.181677 26955 ts_tablet_manager.cc:1428] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
W20260619 15:00:20.192229 26809 tablet.cc:2369] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:00:20.265834 26669 tablet.cc:2369] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:00:20.281395 26811 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:20.281615 26811 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.282608 26811 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951)
I20260619 15:00:20.287755 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.282985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:55175 (local address 127.10.71.129:36951)
0619 15:00:20.283122 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.283126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.283141 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:00:20.283272 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.283274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.283455 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.283553 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:20.283875 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.284667 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.286386 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.287022 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.287135 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.287284 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.287537 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.287612 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 15:00:20.287755 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.282913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:57399 (local address 127.10.71.131:40341)
0619 15:00:20.283024 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.283029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.283436 (+ 407us) server_negotiation.cc:407] Connection header received
0619 15:00:20.283492 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.283498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.283552 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.283626 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:20.283882 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.284667 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.286386 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.286993 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.287136 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.287185 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.287537 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.287612 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:00:20.287809 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.282811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36951 (local address 127.10.71.130:55175)
0619 15:00:20.282950 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:20.282966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:20.283112 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:20.283567 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:20.283570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:20.283583 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:20.283739 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.283748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.284778 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.284781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.286264 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.286274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.287134 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.287138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.287313 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.287400 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 15:00:20.287509 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 15:00:20.287637 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":200}
I20260619 15:00:20.287833 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.282811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:40341 (local address 127.10.71.130:57399)
0619 15:00:20.283253 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:20.283264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:20.283333 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:20.283647 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:20.283649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:20.283660 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:20.283779 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.283788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.284778 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.284781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.286264 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:20.286274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.287138 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.287141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:20.287316 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.287406 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 15:00:20.287516 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 15:00:20.287637 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":408,"thread_start_us":142,"threads_started":1}
I20260619 15:00:20.288398 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec6d8c7c5b347d3bcccc138b84a1e24" candidate_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef20d88415141dfad7d62f547cffb7b" is_pre_election: true
I20260619 15:00:20.288446 26897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec6d8c7c5b347d3bcccc138b84a1e24" candidate_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22b25bed87141faac0d89c52e4e2d95" is_pre_election: true
I20260619 15:00:20.288679 26622 raft_consensus.cc:2454] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5594b20150644e3b4ed8f8295c5ebbf in term 0.
I20260619 15:00:20.288725 26897 raft_consensus.cc:2454] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5594b20150644e3b4ed8f8295c5ebbf in term 0.
I20260619 15:00:20.289193 26693 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b, c22b25bed87141faac0d89c52e4e2d95, f5594b20150644e3b4ed8f8295c5ebbf; no voters:
I20260619 15:00:20.289479 26811 raft_consensus.cc:2781] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:00:20.289565 26811 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:00:20.289605 26811 raft_consensus.cc:3037] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:20.290570 26811 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.290884 26811 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [CANDIDATE]: Term 1 election: Requested vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951)
I20260619 15:00:20.291189 26897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec6d8c7c5b347d3bcccc138b84a1e24" candidate_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22b25bed87141faac0d89c52e4e2d95"
I20260619 15:00:20.291196 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec6d8c7c5b347d3bcccc138b84a1e24" candidate_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef20d88415141dfad7d62f547cffb7b"
I20260619 15:00:20.291327 26897 raft_consensus.cc:3037] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:20.291337 26622 raft_consensus.cc:3037] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:20.292214 26622 raft_consensus.cc:2454] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5594b20150644e3b4ed8f8295c5ebbf in term 1.
I20260619 15:00:20.292299 26897 raft_consensus.cc:2454] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5594b20150644e3b4ed8f8295c5ebbf in term 1.
I20260619 15:00:20.292518 26694 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b, f5594b20150644e3b4ed8f8295c5ebbf; no voters:
I20260619 15:00:20.292724 26811 raft_consensus.cc:2781] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:00:20.292975 26811 raft_consensus.cc:686] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 LEADER]: Becoming Leader. State: Replica: f5594b20150644e3b4ed8f8295c5ebbf, State: Running, Role: LEADER
I20260619 15:00:20.293236 26811 consensus_queue.cc:237] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:20.296172 26477 catalog_manager.cc:5495] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf reported cstate change: term changed from 0 to 1, leader changed from <none> to f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130). New cstate: current_term: 1 leader_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } health_report { overall_health: HEALTHY } } }
I20260619 15:00:20.313639 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.310331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.10.71.130:37541)
0619 15:00:20.310435 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:20.310441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:20.310471 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:00:20.310642 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:20.310644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:20.310687 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:20.310782 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:00:20.310993 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.311458 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.312327 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:20.312553 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:20.312652 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:20.312925 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:00:20.313278 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:00:20.313399 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:20.313461 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:00:20.313507 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 15:00:20.316881 26897 raft_consensus.cc:1261] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Refusing update from remote peer f5594b20150644e3b4ed8f8295c5ebbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:00:20.316922 26622 raft_consensus.cc:1261] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Refusing update from remote peer f5594b20150644e3b4ed8f8295c5ebbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:00:20.317512 26811 consensus_queue.cc:1035] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:00:20.317662 26959 consensus_queue.cc:1035] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:00:20.323611 26962 mvcc.cc:205] Tried to move back new op lower bound from 7298585478412718080 to 7298585478323716096. Current Snapshot: MvccSnapshot[applied={T|T < 7298585478412718080}]
I20260619 15:00:20.330132 26477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:00:20.330272 26477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:00:20.332249 26477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Sending DeleteTablet for 3 replicas of tablet 8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:20.332787 26877 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:20 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:38576
I20260619 15:00:20.332819 26602 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:20 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:58714
I20260619 15:00:20.332919 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26517
I20260619 15:00:20.333285 26741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:20 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:34892
I20260619 15:00:20.333446 26814 tablet_replica.cc:330] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: stopping tablet replica
I20260619 15:00:20.333554 26814 raft_consensus.cc:2229] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:00:20.333808 26814 raft_consensus.cc:2258] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:20.335017 26814 ts_tablet_manager.cc:1905] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:20.336498 26971 tablet_replica.cc:330] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: stopping tablet replica
I20260619 15:00:20.336802 26971 raft_consensus.cc:2229] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:00:20.337037 26814 ts_tablet_manager.cc:1918] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:20.337090 26971 raft_consensus.cc:2258] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:20.337117 26814 log.cc:1196] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:20.337468 26814 ts_tablet_manager.cc:1939] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P f5594b20150644e3b4ed8f8295c5ebbf: Deleting consensus metadata
I20260619 15:00:20.338100 26464 catalog_manager.cc:4841] TS f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 (table test-table [id=9f58b3bfdc554de69beb3d2834657a96]) successfully deleted
I20260619 15:00:20.338474 26971 ts_tablet_manager.cc:1905] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:20.340945 26971 ts_tablet_manager.cc:1918] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:20.341041 26971 log.cc:1196] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:20.341390 26971 ts_tablet_manager.cc:1939] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P c22b25bed87141faac0d89c52e4e2d95: Deleting consensus metadata
W20260619 15:00:20.341887 26464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:36951 (error 108)
I20260619 15:00:20.342409 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26672
I20260619 15:00:20.342487 26463 catalog_manager.cc:4841] TS c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): tablet 8ec6d8c7c5b347d3bcccc138b84a1e24 (table test-table [id=9f58b3bfdc554de69beb3d2834657a96]) successfully deleted
I20260619 15:00:20.344532 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:20.344258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36951 (local address 127.0.0.1:58728)
0619 15:00:20.344415 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:20.344472 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 15:00:20.344666 26464 catalog_manager.cc:4584] TS 2ef20d88415141dfad7d62f547cffb7b (127.10.71.129:36951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ec6d8c7c5b347d3bcccc138b84a1e24: Network error: Client connection negotiation failed: client connection to 127.10.71.129:36951: connect: Connection refused (error 111)
I20260619 15:00:20.348043 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26812
I20260619 15:00:20.353088 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 26446
I20260619 15:00:40.358875 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38943
--webserver_interface=127.10.71.190
--webserver_port=39127
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:40.455168 26973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:40.455359 26973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:40.455392 26973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:40.459473 26973 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 15:00:40.459517 26973 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 15:00:40.459534 26973 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 15:00:40.459548 26973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:00:40.459594 26973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:40.459628 26973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:40.459649 26973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:00:40.459684 26973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:00:40.464155 26973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=39127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:40.465018 26973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:40.465819 26973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:40.473524 26979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:40.473577 26981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:40.473640 26973 server_base.cc:1034] running on GCE node
W20260619 15:00:40.473544 26978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:40.474184 26973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:40.474980 26973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:40.476140 26973 hybrid_clock.cc:648] HybridClock initialized: now 1781881240476126 us; error 25 us; skew 500 ppm
Jun 19 15:00:40 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:40.482748 26973 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:00:40.483515 26973 webserver.cc:460] Webserver started at http://127.10.71.190:39127/ using document root <none> and password file <none>
I20260619 15:00:40.483915 26973 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:40.483963 26973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:40.487036 26973 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:40.489008 26988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:40.490141 26973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:40.490267 26973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "85a80d08786b49a4a2d0b1ae027407f3"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "1bab2407a41a0c9cb3157d986f5ac344"
server_key_iv: "d45c5dd3386641ed89b6263be3635260"
server_key_version: "encryptionkey@0"
I20260619 15:00:40.490751 26973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:40.510370 26973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:40.512905 26973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:40.513119 26973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:40.519435 26973 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:38943
I20260619 15:00:40.519696 27040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:38943 every 8 connection(s)
I20260619 15:00:40.520541 26973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:00:40.524034 27041 sys_catalog.cc:263] Verifying existing consensus state
I20260619 15:00:40.524703 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 26973
I20260619 15:00:40.525166 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:36951
--local_ip_for_outbound_sockets=127.10.71.129
--tserver_master_addrs=127.10.71.190:38943
--webserver_port=46255
--webserver_interface=127.10.71.129
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:00:40.525919 27041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap starting.
I20260619 15:00:40.541884 27041 log.cc:824] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:40.552124 27041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:00:40.552520 27041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Bootstrap complete.
I20260619 15:00:40.555809 27041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:40.556149 27041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a80d08786b49a4a2d0b1ae027407f3, State: Initialized, Role: FOLLOWER
I20260619 15:00:40.556605 27041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:40.556738 27041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:00:40.556820 27041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:00:40.556924 27041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 2 FOLLOWER]: Advancing to term 3
I20260619 15:00:40.558437 27041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:40.558753 27041 leader_election.cc:302] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a80d08786b49a4a2d0b1ae027407f3; no voters:
I20260619 15:00:40.559094 27041 leader_election.cc:288] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 15:00:40.559234 27045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 15:00:40.559520 27045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [term 3 LEADER]: Becoming Leader. State: Replica: 85a80d08786b49a4a2d0b1ae027407f3, State: Running, Role: LEADER
I20260619 15:00:40.559823 27045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } }
I20260619 15:00:40.560241 27041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:00:40.560774 27045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a80d08786b49a4a2d0b1ae027407f3. Latest consensus state: current_term: 3 leader_uuid: "85a80d08786b49a4a2d0b1ae027407f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } } }
I20260619 15:00:40.560878 27045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:00:40.562065 27050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:00:40.562471 27046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85a80d08786b49a4a2d0b1ae027407f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a80d08786b49a4a2d0b1ae027407f3" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38943 } } }
I20260619 15:00:40.562633 27046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:00:40.566818 27050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:00:40.567410 27050 catalog_manager.cc:1259] Loaded cluster ID: 5ec36fd2c1ad4744a914f30badffa6ef
I20260619 15:00:40.567477 27050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:00:40.570036 27050 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:00:40.570837 27050 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Loaded TSK: 0
I20260619 15:00:40.570919 27050 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85a80d08786b49a4a2d0b1ae027407f3: Loaded TSK: 1
F20260619 15:00:40.571616 27050 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcecdb429ad google::LogMessage::Fail() at ??:0
@ 0x7fcecdb4588a google::LogMessage::SendToLog() at ??:0
@ 0x7fcecdb424de google::LogMessage::Flush() at ??:0
@ 0x7fcecdb43a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fced080618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fced07fef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fced84a77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fced848199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fced8482178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fced84c3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fced84e282b std::function<>::operator()() at ??:0
@ 0x7fced8481b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fced8482f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fced847f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fced84c3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcecfd943d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcecfd94c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcecfd963f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcecfd85512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fced13c96db start_thread at ??:0
@ 0x7fcecc64061f clone at ??:0
W20260619 15:00:40.652642 27043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:40.652938 27043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:40.652993 27043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:40.656800 27043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:40.656881 27043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:40.656981 27043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:00:40.661999 27043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:36951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=46255
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:40.662931 27043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:40.663856 27043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:40.672883 27088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:40.673009 27091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:40.673096 27043 server_base.cc:1034] running on GCE node
W20260619 15:00:40.672861 27089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:40.673659 27043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:40.674203 27043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:40.675391 27043 hybrid_clock.cc:648] HybridClock initialized: now 1781881240675368 us; error 39 us; skew 500 ppm
Jun 19 15:00:40 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:40.678095 27043 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 15:00:40.678881 27043 webserver.cc:460] Webserver started at http://127.10.71.129:46255/ using document root <none> and password file <none>
I20260619 15:00:40.679430 27043 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:40.679524 27043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:40.683241 27043 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:40.685254 27098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:40.686401 27043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:40.686534 27043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "2ef20d88415141dfad7d62f547cffb7b"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "10d79e580e9ac03439977b0ebee4d774"
server_key_iv: "42bf510856a1b2f1b58566dfc7cfc16e"
server_key_version: "encryptionkey@0"
I20260619 15:00:40.686995 27043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:40.713585 27043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:40.716595 27043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:40.716794 27043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:40.717348 27043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:40.718412 27105 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 15:00:40.723587 27043 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 15:00:40.723660 27043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 15:00:40.723733 27043 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 15:00:40.725284 27043 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 15:00:40.725353 27043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 15:00:40.725642 27105 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap starting.
I20260619 15:00:40.735351 27043 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:36951
I20260619 15:00:40.735567 27212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:36951 every 8 connection(s)
I20260619 15:00:40.736240 27043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 15:00:40.741616 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27043
I20260619 15:00:40.741994 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:37541
--local_ip_for_outbound_sockets=127.10.71.130
--tserver_master_addrs=127.10.71.190:38943
--webserver_port=39333
--webserver_interface=127.10.71.130
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:00:40.746258 27105 log.cc:824] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:40.752267 27105 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 15:00:40.752642 27105 tablet_bootstrap.cc:492] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Bootstrap complete.
I20260619 15:00:40.753624 27105 ts_tablet_manager.cc:1397] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20260619 15:00:40.755643 27105 raft_consensus.cc:348] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:40.756050 27105 raft_consensus.cc:729] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef20d88415141dfad7d62f547cffb7b, State: Initialized, Role: FOLLOWER
I20260619 15:00:40.756500 27105 consensus_queue.cc:260] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:40.757326 27105 ts_tablet_manager.cc:1428] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 15:00:40.811313 27215 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:00:40.737951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:45191)
0619 15:00:40.738372 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:40.738416 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:40.739105 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:40.810413 (+ 71308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":135,"threads_started":1}
W20260619 15:00:40.812036 27213 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:38943 (0 consecutive failures): Network error: Failed to ping master at 127.10.71.190:38943: Client connection negotiation failed: client connection to 127.10.71.190:38943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:00:40.812868 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:40.812578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:38945)
0619 15:00:40.812735 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:40.812805 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 15:00:40.813699 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:40.813443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:32943)
0619 15:00:40.813599 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:40.813645 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 15:00:40.813916 27213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 15:00:40.863272 27216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:40.863464 27216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:40.863498 27216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:40.866900 27216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:40.866943 27216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:40.866990 27216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:00:40.871695 27216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:37541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=39333
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:40.872560 27216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:40.873399 27216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:40.881737 27226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:40.881744 27224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:40.881754 27223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:40.881948 27216 server_base.cc:1034] running on GCE node
I20260619 15:00:40.882411 27216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:40.882961 27216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:40.884140 27216 hybrid_clock.cc:648] HybridClock initialized: now 1781881240884130 us; error 21 us; skew 500 ppm
Jun 19 15:00:40 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:40.886384 27216 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 15:00:40.887084 27216 webserver.cc:460] Webserver started at http://127.10.71.130:39333/ using document root <none> and password file <none>
I20260619 15:00:40.887492 27216 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:40.887562 27216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:40.891124 27216 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:40.892773 27233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:40.893775 27216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:40.893858 27216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "f5594b20150644e3b4ed8f8295c5ebbf"
format_stamp: "Formatted at 2026-06-19 14:59:58 on dist-test-slave-3qdh"
server_key: "b3a276e74809f39168a7526dfb7c21c4"
server_key_iv: "4427591a71d710bb58a44777fa655678"
server_key_version: "encryptionkey@0"
I20260619 15:00:40.894305 27216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:40.907275 27216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:40.909900 27216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:40.910110 27216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:40.910648 27216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:40.911357 27216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:40.911429 27216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:40.911485 27216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:40.911518 27216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:40.919648 27216 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:37541
I20260619 15:00:40.919668 27346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:37541 every 8 connection(s)
I20260619 15:00:40.920648 27216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:00:40.923993 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:40.922399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:36173)
0619 15:00:40.923125 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:40.923243 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":95,"threads_started":1}
W20260619 15:00:40.924456 27347 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:38943 (0 consecutive failures): Network error: Failed to ping master at 127.10.71.190:38943: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
I20260619 15:00:40.925021 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:40.924791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:55967)
0619 15:00:40.924912 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:40.924960 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 15:00:40.925648 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:40.925424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:49629)
0619 15:00:40.925524 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:40.925566 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 15:00:40.925822 27347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 15:00:40.928427 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27216
I20260619 15:00:40.928844 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:40341
--local_ip_for_outbound_sockets=127.10.71.131
--tserver_master_addrs=127.10.71.190:38943
--webserver_port=42651
--webserver_interface=127.10.71.131
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:41.026166 27350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:41.026366 27350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:41.026400 27350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:41.030195 27350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:41.030243 27350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:41.030298 27350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:00:41.034636 27350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:40341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=42651
--tserver_master_addrs=127.10.71.190:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:41.035435 27350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:41.036252 27350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:41.045310 27358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:41.045296 27356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:41.045341 27350 server_base.cc:1034] running on GCE node
W20260619 15:00:41.045300 27355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:41.045826 27350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:41.046373 27350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:41.047560 27350 hybrid_clock.cc:648] HybridClock initialized: now 1781881241047536 us; error 44 us; skew 500 ppm
Jun 19 15:00:41 dist-test-slave-3qdh krb5kdc[25897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881241, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:41.050143 27350 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 15:00:41.051007 27350 webserver.cc:460] Webserver started at http://127.10.71.131:42651/ using document root <none> and password file <none>
I20260619 15:00:41.051513 27350 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:41.051604 27350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:41.054934 27350 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:00:41.056764 27365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:41.057862 27350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:41.057996 27350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "c22b25bed87141faac0d89c52e4e2d95"
format_stamp: "Formatted at 2026-06-19 14:59:59 on dist-test-slave-3qdh"
server_key: "ab4b6aa67488be2a52df3ea0e8a97757"
server_key_iv: "5f3a64cfdbcd2ade2f085aeb03ffa5b8"
server_key_version: "encryptionkey@0"
I20260619 15:00:41.058451 27350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:41.076155 27350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:41.078881 27350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:41.079059 27350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:41.079607 27350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:41.080420 27350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:41.080504 27350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:41.080578 27350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:41.080621 27350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:41.088994 27350 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:40341
I20260619 15:00:41.089023 27478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:40341 every 8 connection(s)
I20260619 15:00:41.089871 27350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 15:00:41.093103 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:41.091764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:40237)
0619 15:00:41.092406 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:41.092502 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":94,"threads_started":1}
W20260619 15:00:41.093533 27479 heartbeater.cc:644] Failed to heartbeat to 127.10.71.190:38943 (0 consecutive failures): Network error: Failed to ping master at 127.10.71.190:38943: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
I20260619 15:00:41.094064 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27350
I20260619 15:00:41.094094 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:41.093852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:41653)
0619 15:00:41.093965 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:41.094032 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 15:00:41.094648 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:41.094425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:42019)
0619 15:00:41.094533 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:41.094580 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 15:00:41.094818 27479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 15:00:41.814798 27483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:41.814410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:53979)
0619 15:00:41.814637 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:41.814705 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260619 15:00:41.926844 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:41.926385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:36399)
0619 15:00:41.926639 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:41.926699 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":60,"threads_started":1}
I20260619 15:00:42.095636 27485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.095284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:39727)
0619 15:00:42.095499 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:42.095542 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260619 15:00:42.400002 27486 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:42.400230 27486 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:42.401456 27486 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
Jun 19 15:00:42 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.131@KRBTEST.COM
Jun 19 15:00:42 dist-test-slave-3qdh krb5kdc[25897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.130@KRBTEST.COM
I20260619 15:00:42.414664 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.401860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:53553 (local address 127.10.71.130:37541)
0619 15:00:42.402236 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:42.402255 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:42.402285 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:00:42.402992 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:42.402999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:42.403273 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:42.403560 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:42.404832 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.405714 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.406897 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.407358 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.407546 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:42.410622 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:42.410626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:42.410635 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:42.410659 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:42.412299 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:42.412811 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:42.412818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:42.412824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:42.412933 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:42.413289 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:42.413292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:42.413294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:42.413685 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:42.413778 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:42.414276 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 15:00:42.414491 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":89,"threads_started":1}
I20260619 15:00:42.414654 27487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.401596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:40341 (local address 127.10.71.129:55777)
0619 15:00:42.401878 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:42.401893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:42.402007 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:42.403581 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:42.403586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:42.404153 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:42.404661 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:42.404675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.405877 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.405881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:42.406778 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:42.406788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.407490 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.407493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:42.407689 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:42.408486 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:42.408498 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:42.410100 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:42.412335 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:42.412340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:42.412354 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:42.412707 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:42.413080 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:42.413082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:42.413084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:42.413190 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:42.413877 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:42.413879 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:42.414009 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 15:00:42.414171 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 15:00:42.414483 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":207,"spinlock_wait_cycles":27008,"thread_start_us":74,"threads_started":1}
I20260619 15:00:42.414726 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.401796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:55777 (local address 127.10.71.131:40341)
0619 15:00:42.402217 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:42.402243 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:42.402286 (+ 43us) server_negotiation.cc:407] Connection header received
0619 15:00:42.402915 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:42.402920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:42.403238 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:42.403512 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:42.404825 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.405708 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.406941 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.407360 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.407539 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:42.410330 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:42.410334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:42.410344 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:42.410370 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:42.412181 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:42.412843 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:42.412848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:42.412851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:42.412950 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:42.413422 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:42.413425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:42.413426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:42.413771 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:42.413886 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:42.414289 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 15:00:42.414551 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":103,"threads_started":1}
I20260619 15:00:42.415297 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.401717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37541 (local address 127.10.71.129:53553)
0619 15:00:42.402046 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:42.402089 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:42.402181 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:42.403580 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:42.403586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:42.404135 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:42.404660 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:42.404674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.405876 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.405879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:42.406775 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:42.406783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:42.407489 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:42.407492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:42.407682 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:42.408409 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:42.408425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:42.410427 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:42.412457 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:42.412459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:42.412460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:42.412698 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:42.413044 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:42.413046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:42.413048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:42.413186 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:42.413813 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:42.413821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:42.414017 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 15:00:42.414171 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 15:00:42.414565 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":236,"spinlock_wait_cycles":11904,"thread_start_us":74,"threads_started":1}
W20260619 15:00:42.417042 27099 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
W20260619 15:00:42.417395 27100 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:42.417521 27100 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b; no voters: c22b25bed87141faac0d89c52e4e2d95, f5594b20150644e3b4ed8f8295c5ebbf
I20260619 15:00:42.417765 27486 raft_consensus.cc:2726] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 15:00:42.815810 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.815500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:38499)
0619 15:00:42.815628 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:42.815682 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 15:00:42.927946 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:42.927534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:33743)
0619 15:00:42.927820 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:42.927868 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260619 15:00:43.096661 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:43.096258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:48421)
0619 15:00:43.096529 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:43.096578 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":119,"threads_started":1}
I20260619 15:00:43.817023 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:43.816636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:37505)
0619 15:00:43.816861 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:43.816912 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260619 15:00:43.929064 27494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:43.928653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:42403)
0619 15:00:43.928912 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:43.928962 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260619 15:00:44.097726 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:44.097328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:37617)
0619 15:00:44.097575 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:44.097628 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260619 15:00:44.465519 27496 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:44.465701 27496 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:44.466149 27496 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
W20260619 15:00:44.466974 27099 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
W20260619 15:00:44.467094 27100 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:44.467156 27100 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b; no voters: c22b25bed87141faac0d89c52e4e2d95, f5594b20150644e3b4ed8f8295c5ebbf
I20260619 15:00:44.467304 27496 raft_consensus.cc:2726] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 15:00:44.818228 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:44.817762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:35199)
0619 15:00:44.818048 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:44.818113 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260619 15:00:44.930104 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:44.929718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:55497)
0619 15:00:44.929966 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:44.930022 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260619 15:00:45.098776 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:45.098365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:51585)
0619 15:00:45.098615 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:45.098659 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
I20260619 15:00:45.819638 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:45.819236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:40039)
0619 15:00:45.819500 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:45.819547 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
I20260619 15:00:45.931192 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:45.930826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:38673)
0619 15:00:45.931064 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:45.931111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260619 15:00:46.099752 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:46.099377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:47933)
0619 15:00:46.099609 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:46.099656 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260619 15:00:46.698195 27503 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:46.698333 27503 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:46.698796 27503 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
W20260619 15:00:46.699525 27100 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
W20260619 15:00:46.699620 27099 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:46.699797 27099 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b; no voters: c22b25bed87141faac0d89c52e4e2d95, f5594b20150644e3b4ed8f8295c5ebbf
I20260619 15:00:46.699941 27503 raft_consensus.cc:2726] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 15:00:46.820750 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:46.820367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:35161)
0619 15:00:46.820608 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:46.820658 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260619 15:00:46.932241 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:46.931840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:33757)
0619 15:00:46.932097 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:46.932148 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260619 15:00:47.100822 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:47.100458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:47257)
0619 15:00:47.100682 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:47.100738 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260619 15:00:47.821841 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:47.821471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:39751)
0619 15:00:47.821706 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:47.821752 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260619 15:00:47.933561 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:47.933054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:56323)
0619 15:00:47.933416 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:47.933475 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":85,"threads_started":1}
I20260619 15:00:48.101925 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:48.101501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:33971)
0619 15:00:48.101774 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:48.101828 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260619 15:00:48.822875 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:48.822510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:35355)
0619 15:00:48.822739 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:48.822786 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260619 15:00:48.934695 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:48.934285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:51905)
0619 15:00:48.934521 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:48.934588 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260619 15:00:49.103009 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:49.102678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:46477)
0619 15:00:49.102889 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:49.102934 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260619 15:00:49.139369 27513 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:49.139487 27513 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:49.139942 27513 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
W20260619 15:00:49.140725 27100 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
W20260619 15:00:49.140880 27099 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:49.140954 27099 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b; no voters: c22b25bed87141faac0d89c52e4e2d95, f5594b20150644e3b4ed8f8295c5ebbf
I20260619 15:00:49.141094 27513 raft_consensus.cc:2726] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 15:00:49.824056 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:49.823603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:45833)
0619 15:00:49.823892 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:49.823955 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260619 15:00:49.935835 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:49.935368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:49141)
0619 15:00:49.935674 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:49.935730 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
I20260619 15:00:50.104111 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:50.103698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:42233)
0619 15:00:50.103981 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:50.104029 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20260619 15:00:50.825253 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:50.824857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:57713)
0619 15:00:50.825094 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:50.825159 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
I20260619 15:00:50.937031 27518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:50.936666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:59821)
0619 15:00:50.936892 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:50.936943 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260619 15:00:51.105162 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:51.104780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:51247)
0619 15:00:51.105026 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:51.105078 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":63,"threads_started":1}
I20260619 15:00:51.545949 27520 raft_consensus.cc:482] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:51.546072 27520 raft_consensus.cc:504] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22b25bed87141faac0d89c52e4e2d95" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 40341 } } peers { permanent_uuid: "2ef20d88415141dfad7d62f547cffb7b" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36951 } } peers { permanent_uuid: "f5594b20150644e3b4ed8f8295c5ebbf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37541 } }
I20260619 15:00:51.546535 27520 leader_election.cc:288] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341), f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541)
W20260619 15:00:51.547386 27099 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c22b25bed87141faac0d89c52e4e2d95 (127.10.71.131:40341): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
W20260619 15:00:51.547479 27100 leader_election.cc:341] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5594b20150644e3b4ed8f8295c5ebbf (127.10.71.130:37541): Not found: Tablet not found: 8ec6d8c7c5b347d3bcccc138b84a1e24
I20260619 15:00:51.547590 27100 leader_election.cc:302] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef20d88415141dfad7d62f547cffb7b; no voters: c22b25bed87141faac0d89c52e4e2d95, f5594b20150644e3b4ed8f8295c5ebbf
I20260619 15:00:51.547760 27520 raft_consensus.cc:2726] T 8ec6d8c7c5b347d3bcccc138b84a1e24 P 2ef20d88415141dfad7d62f547cffb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 15:00:51.826438 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:51.826023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:46481)
0619 15:00:51.826273 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:51.826330 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260619 15:00:51.938225 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:51.937831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:35371)
0619 15:00:51.938074 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:51.938133 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260619 15:00:52.106365 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:52.105974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:39373)
0619 15:00:52.106222 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:52.106278 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260619 15:00:52.827713 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:52.827244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:55261)
0619 15:00:52.827537 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:52.827602 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
I20260619 15:00:52.939252 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:52.938851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:38997)
0619 15:00:52.939118 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:52.939167 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260619 15:00:53.107439 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:53.107064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:41357)
0619 15:00:53.107305 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:53.107353 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260619 15:00:53.828878 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:53.828498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:40349)
0619 15:00:53.828732 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:53.828785 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260619 15:00:53.940259 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:53.939887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:50823)
0619 15:00:53.940120 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:53.940169 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260619 15:00:54.108575 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:54.108099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:44641)
0619 15:00:54.108412 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:54.108479 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260619 15:00:54.829929 27530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:54.829565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:39721)
0619 15:00:54.829785 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:54.829836 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260619 15:00:54.941342 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:54.940999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:52415)
0619 15:00:54.941216 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:54.941264 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260619 15:00:55.109591 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:55.109220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.131:46767)
0619 15:00:55.109457 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:55.109506 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260619 15:00:55.831133 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:55.830758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.129:55851)
0619 15:00:55.830992 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:55.831041 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260619 15:00:55.942489 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:55.942120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38943 (local address 127.10.71.130:49129)
0619 15:00:55.942358 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:55.942404 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.190:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260619 15:00:56.098704 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27043
I20260619 15:00:56.104352 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27216
I20260619 15:00:56.109565 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27350
2026-06-19T15:00:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58828 ms)
[----------] 30 tests from SecurityITest (227351 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:00:56 dist-test-slave-3qdh krb5kdc[27537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:00:56 dist-test-slave-3qdh krb5kdc[27537](info): set up 2 sockets
Jun 19 15:00:56 dist-test-slave-3qdh krb5kdc[27537](info): commencing operation
krb5kdc: starting...
W20260619 15:00:57.131881 10526 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:00:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:00:57.244763 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37387
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:57.345758 27550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:57.345981 27550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:57.346019 27550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:57.349714 27550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:00:57.349776 27550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:57.349795 27550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:57.349812 27550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:00:57.349860 27550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:00:57.354470 27550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:57.355281 27550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:57.356118 27550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:00:57.363854 27550 server_base.cc:1034] running on GCE node
W20260619 15:00:57.363883 27555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.363804 27558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.363865 27556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:57.364609 27550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:57.365572 27550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:57.366753 27550 hybrid_clock.cc:648] HybridClock initialized: now 1781881257366738 us; error 31 us; skew 500 ppm
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:57.369570 27550 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:00:57.370360 27550 webserver.cc:460] Webserver started at http://127.10.71.190:39821/ using document root <none> and password file <none>
I20260619 15:00:57.370880 27550 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:57.370970 27550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:57.371207 27550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:57.373162 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "17ed192e2207425b9ab175d1036d157d"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "bf515284dac64ff90a70985359c1a12f"
server_key_iv: "fea7c918638b7b5f4344280f4961b34b"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.373677 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "17ed192e2207425b9ab175d1036d157d"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "bf515284dac64ff90a70985359c1a12f"
server_key_iv: "fea7c918638b7b5f4344280f4961b34b"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.377247 27550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:00:57.379307 27565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.380518 27550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:00:57.380658 27550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "17ed192e2207425b9ab175d1036d157d"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "bf515284dac64ff90a70985359c1a12f"
server_key_iv: "fea7c918638b7b5f4344280f4961b34b"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.380761 27550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:57.397774 27550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:57.400097 27550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:57.400257 27550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:57.406581 27550 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:37387
I20260619 15:00:57.406602 27617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:37387 every 8 connection(s)
I20260619 15:00:57.407331 27550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:00:57.409603 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27550
I20260619 15:00:57.409713 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:00:57.409787 27618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:57.409991 10526 external_mini_cluster.cc:1428] Setting key 957b78aef0ec65d3205ab27973eb8b05
I20260619 15:00:57.415292 27618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: Bootstrap starting.
I20260619 15:00:57.417649 27618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: Neither blocks nor log segments found. Creating new log.
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:00:57.418320 27618 log.cc:824] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:57.419919 27618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: No bootstrap required, opened a new log
I20260619 15:00:57.422610 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:57.411209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.10.71.190:37387)
0619 15:00:57.411615 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:57.411628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:57.411665 (+ 37us) server_negotiation.cc:407] Connection header received
0619 15:00:57.412245 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:57.412259 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:57.412597 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:57.412891 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:57.413622 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.414477 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.415336 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.415862 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.416033 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:57.418537 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:57.418542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:57.418552 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:57.418575 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:57.420126 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:57.420588 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:57.420592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:57.420595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:57.420714 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:57.421000 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:57.421002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:57.421003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:57.421332 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:57.421464 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:57.421731 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 15:00:57.421987 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260619 15:00:57.423712 27618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ed192e2207425b9ab175d1036d157d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37387 } }
I20260619 15:00:57.423942 27618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:57.423985 27618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ed192e2207425b9ab175d1036d157d, State: Initialized, Role: FOLLOWER
I20260619 15:00:57.424391 27618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ed192e2207425b9ab175d1036d157d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37387 } }
I20260619 15:00:57.424546 27618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:00:57.424612 27618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:00:57.424727 27618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:57.425451 27618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ed192e2207425b9ab175d1036d157d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37387 } }
I20260619 15:00:57.425756 27618 leader_election.cc:302] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17ed192e2207425b9ab175d1036d157d; no voters:
I20260619 15:00:57.426044 27618 leader_election.cc:288] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:00:57.426151 27623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:00:57.426342 27623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [term 1 LEADER]: Becoming Leader. State: Replica: 17ed192e2207425b9ab175d1036d157d, State: Running, Role: LEADER
I20260619 15:00:57.426633 27623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ed192e2207425b9ab175d1036d157d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37387 } }
I20260619 15:00:57.427052 27618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:00:57.428117 27624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17ed192e2207425b9ab175d1036d157d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ed192e2207425b9ab175d1036d157d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37387 } } }
I20260619 15:00:57.428242 27624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [sys.catalog]: This master's current role is: LEADER
I20260619 15:00:57.428546 27625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17ed192e2207425b9ab175d1036d157d. Latest consensus state: current_term: 1 leader_uuid: "17ed192e2207425b9ab175d1036d157d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ed192e2207425b9ab175d1036d157d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37387 } } }
I20260619 15:00:57.428704 27625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d [sys.catalog]: This master's current role is: LEADER
W20260619 15:00:57.431340 27639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:00:57.431402 27639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:00:57.431479 27632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:00:57.432160 27632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:00:57.436609 27632 catalog_manager.cc:1347] Generated new cluster ID: c0514372e19a44169f76e249edf76115
I20260619 15:00:57.436678 27632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:00:57.445778 27632 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:00:57.446780 27632 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:00:57.457755 27632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: Generated new TSK 0
I20260619 15:00:57.458559 27632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 15:00:57.483726 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37387
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:57.585088 27644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:57.585321 27644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:57.585389 27644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:57.589138 27644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:57.589212 27644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:57.589306 27644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:00:57.594169 27644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:57.595029 27644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:57.595971 27644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:57.604467 27652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.604470 27649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.604509 27650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:57.604705 27644 server_base.cc:1034] running on GCE node
I20260619 15:00:57.605001 27644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:57.605497 27644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:57.606643 27644 hybrid_clock.cc:648] HybridClock initialized: now 1781881257606621 us; error 33 us; skew 500 ppm
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:57.608938 27644 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 15:00:57.609583 27644 webserver.cc:460] Webserver started at http://127.10.71.129:40785/ using document root <none> and password file <none>
I20260619 15:00:57.610034 27644 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:57.610091 27644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:57.610234 27644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:57.611945 27644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "2c690e12c5eb41d3b8d9873d5db08f9c"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "e82a3441a9dbbc2b86f589f77cf1b601"
server_key_iv: "0fe43d50032eeba8f4505e8d86d26ab4"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.612413 27644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "2c690e12c5eb41d3b8d9873d5db08f9c"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "e82a3441a9dbbc2b86f589f77cf1b601"
server_key_iv: "0fe43d50032eeba8f4505e8d86d26ab4"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.615381 27644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:00:57.617249 27659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.618263 27644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 15:00:57.618377 27644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "2c690e12c5eb41d3b8d9873d5db08f9c"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "e82a3441a9dbbc2b86f589f77cf1b601"
server_key_iv: "0fe43d50032eeba8f4505e8d86d26ab4"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.618468 27644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:57.626744 27644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:57.629431 27644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:57.629616 27644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:57.630115 27644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:57.630847 27644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:57.630913 27644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.630977 27644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:57.631023 27644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.638916 27644 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:39061
I20260619 15:00:57.638960 27772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:39061 every 8 connection(s)
I20260619 15:00:57.639689 27644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:00:57.648867 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27644
I20260619 15:00:57.649021 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 15:00:57.649288 10526 external_mini_cluster.cc:1428] Setting key c2001e6b83f19601acdfa3dd56db9c2b
I20260619 15:00:57.652033 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:57.641265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:46773 (local address 127.10.71.190:37387)
0619 15:00:57.641410 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:57.641415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:57.642130 (+ 715us) server_negotiation.cc:407] Connection header received
0619 15:00:57.643279 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:57.643281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:57.643326 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:57.643396 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:57.644502 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.645046 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.646096 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.646305 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.646377 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:57.648644 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:57.648648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:57.648650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:57.648685 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:57.650082 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:57.650594 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:57.650598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:57.650600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:57.650680 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:57.651007 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:57.651012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:57.651014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:57.651186 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:57.651295 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:57.651745 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 15:00:57.651871 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 15:00:57.652782 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:57.641576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37387 (local address 127.10.71.129:46773)
0619 15:00:57.641973 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:57.642025 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:57.643111 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:57.643545 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:57.643552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:57.643842 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:57.644309 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:57.644320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.645152 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.645155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:57.645996 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:57.646005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.646419 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.646421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:57.646548 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:57.647086 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:57.647102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:57.648488 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:57.650214 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:57.650220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:57.650232 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:57.650474 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:57.650779 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:57.650782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:57.650784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:57.650913 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:57.651307 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:57.651315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:57.651525 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 15:00:57.651705 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 15:00:57.651991 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260619 15:00:57.653900 27773 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37387
I20260619 15:00:57.654117 27773 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:57.654635 27773 heartbeater.cc:511] Master 127.10.71.190:37387 requested a full tablet report, sending...
I20260619 15:00:57.656060 27580 ts_manager.cc:194] Registered new tserver with Master: 2c690e12c5eb41d3b8d9873d5db08f9c (127.10.71.129:39061)
I20260619 15:00:57.657158 27580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:46773
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 15:00:57.674758 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37387
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:57.776612 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:57.776906 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:57.776983 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:57.780922 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:57.781006 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:57.781128 27778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:00:57.786382 27778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:57.787293 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:57.788228 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:57.796990 27786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.796990 27783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:57.797195 27778 server_base.cc:1034] running on GCE node
W20260619 15:00:57.797016 27784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:57.797753 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:57.798322 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:57.799475 27778 hybrid_clock.cc:648] HybridClock initialized: now 1781881257799454 us; error 36 us; skew 500 ppm
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:57.801992 27778 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 15:00:57.802704 27778 webserver.cc:460] Webserver started at http://127.10.71.130:34405/ using document root <none> and password file <none>
I20260619 15:00:57.803143 27778 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:57.803205 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:57.803335 27778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:57.805006 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "e95b077931c149f8aeeb57dd49efaddf"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "e8d0b2b36645a97bae790d373b820daa"
server_key_iv: "288120a813e4b88532a1c835d6b6ec7c"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.805428 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "e95b077931c149f8aeeb57dd49efaddf"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "e8d0b2b36645a97bae790d373b820daa"
server_key_iv: "288120a813e4b88532a1c835d6b6ec7c"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.808461 27778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:00:57.810650 27793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.811738 27778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:57.811874 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "e95b077931c149f8aeeb57dd49efaddf"
format_stamp: "Formatted at 2026-06-19 15:00:57 on dist-test-slave-3qdh"
server_key: "e8d0b2b36645a97bae790d373b820daa"
server_key_iv: "288120a813e4b88532a1c835d6b6ec7c"
server_key_version: "encryptionkey@0"
I20260619 15:00:57.811972 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:57.833130 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:57.835690 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:57.835863 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:57.836370 27778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:57.837039 27778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:57.837105 27778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.837163 27778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:57.837209 27778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:57.845639 27778 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:39651
I20260619 15:00:57.845676 27906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:39651 every 8 connection(s)
I20260619 15:00:57.846452 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:00:57.849865 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27778
I20260619 15:00:57.850000 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 15:00:57.850301 10526 external_mini_cluster.cc:1428] Setting key c2fa98994c6f83518453271d11a82780
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 15:00:57.859826 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:57.848118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:55205 (local address 127.10.71.190:37387)
0619 15:00:57.848239 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:57.848242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:57.848995 (+ 753us) server_negotiation.cc:407] Connection header received
0619 15:00:57.850389 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:57.850395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:57.850462 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:57.850541 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:57.851607 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.852134 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.853464 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.853683 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.853766 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:57.856260 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:57.856263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:57.856265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:57.856286 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:57.857723 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:57.858328 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:57.858331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:57.858332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:57.858404 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:57.858774 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:57.858776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:57.858778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:57.858933 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:57.859026 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:57.859627 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 15:00:57.859716 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 15:00:57.860706 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:57.848446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37387 (local address 127.10.71.130:55205)
0619 15:00:57.848838 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:57.848874 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:57.850146 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:57.850732 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:57.850745 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:57.851038 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:57.851460 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:57.851473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.852261 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.852264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:57.853333 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:57.853347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:57.853832 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:57.853837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:57.854125 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:57.854743 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:57.854759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:57.856089 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:57.857836 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:57.857843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:57.857858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:57.858224 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:57.858496 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:57.858502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:57.858507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:57.858676 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:57.859070 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:57.859077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:57.859382 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 15:00:57.859581 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 15:00:57.859889 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260619 15:00:57.861804 27907 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37387
I20260619 15:00:57.862040 27907 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 15:00:57.862608 27907 heartbeater.cc:511] Master 127.10.71.190:37387 requested a full tablet report, sending...
I20260619 15:00:57.863610 27580 ts_manager.cc:194] Registered new tserver with Master: e95b077931c149f8aeeb57dd49efaddf (127.10.71.130:39651)
I20260619 15:00:57.864441 27580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:55205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 15:00:57.878379 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37387
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:00:57.978468 27912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:57.978674 27912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:57.978709 27912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:57.982391 27912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:57.982434 27912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:57.982642 27912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:00:57.989063 27912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:57.990025 27912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:57.990968 27912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:57.999532 27920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.999577 27917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:57.999600 27918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:57.999673 27912 server_base.cc:1034] running on GCE node
I20260619 15:00:58.000164 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:58.000708 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:58.001909 27912 hybrid_clock.cc:648] HybridClock initialized: now 1781881258001878 us; error 48 us; skew 500 ppm
Jun 19 15:00:57 dist-test-slave-3qdh krb5kdc[27537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:00:58.004274 27912 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 15:00:58.005050 27912 webserver.cc:460] Webserver started at http://127.10.71.131:46829/ using document root <none> and password file <none>
I20260619 15:00:58.005511 27912 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:58.005561 27912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:58.005697 27912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:58.007539 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "e1a93046a56440c783a7e6de8141bf4e"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "2a3a1585b529f62880f8bcbba3676eda"
server_key_iv: "db99c981e189f044b4e230289a2dbb9b"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.008029 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "e1a93046a56440c783a7e6de8141bf4e"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "2a3a1585b529f62880f8bcbba3676eda"
server_key_iv: "db99c981e189f044b4e230289a2dbb9b"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.011379 27912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 15:00:58.013528 27927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.014552 27912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:00:58.014633 27912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "e1a93046a56440c783a7e6de8141bf4e"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "2a3a1585b529f62880f8bcbba3676eda"
server_key_iv: "db99c981e189f044b4e230289a2dbb9b"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.014710 27912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:58.022899 27912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:58.025272 27912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:58.025395 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:58.025805 27912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:58.026456 27912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:58.026499 27912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.026533 27912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:58.026547 27912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.034778 27912 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:43907
I20260619 15:00:58.034839 28040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:43907 every 8 connection(s)
I20260619 15:00:58.035518 27912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
Jun 19 15:00:58 dist-test-slave-3qdh krb5kdc[27537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:00:58.043507 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 27912
I20260619 15:00:58.043625 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 15:00:58.043908 10526 external_mini_cluster.cc:1428] Setting key 00103faf9f03dc02aad29691894d44f0
I20260619 15:00:58.046957 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.037034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:57879 (local address 127.10.71.190:37387)
0619 15:00:58.037159 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.037163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.037830 (+ 667us) server_negotiation.cc:407] Connection header received
0619 15:00:58.038832 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.038835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.038876 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.038939 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:58.039819 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.040274 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.041348 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.041556 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.041631 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.043717 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:58.043721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:58.043723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:58.043749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:58.045192 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:58.045628 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:58.045631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:58.045632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:58.045705 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:58.046019 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:58.046022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:58.046024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:58.046178 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:58.046275 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.046683 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.046793 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 15:00:58.047554 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.037318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:37387 (local address 127.10.71.131:57879)
0619 15:00:58.037684 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.037718 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.038664 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.039071 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.039076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.039337 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:58.039710 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.039720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.040391 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.040393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.041241 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.041248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.041677 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.041680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.041809 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.042259 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:00:58.042273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:58.043512 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:58.045301 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:58.045305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:58.045316 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:58.045536 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:58.045785 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:00:58.045806 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:00:58.045807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:00:58.045926 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:00:58.046274 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:58.046279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:58.046478 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.046643 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.046921 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260619 15:00:58.048508 28041 heartbeater.cc:347] Connected to a master server at 127.10.71.190:37387
I20260619 15:00:58.048666 28041 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:58.049038 28041 heartbeater.cc:511] Master 127.10.71.190:37387 requested a full tablet report, sending...
I20260619 15:00:58.049853 27580 ts_manager.cc:194] Registered new tserver with Master: e1a93046a56440c783a7e6de8141bf4e (127.10.71.131:43907)
I20260619 15:00:58.050335 27580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:57879
I20260619 15:00:58.057632 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:00:58.065166 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.059315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.10.71.190:37387)
0619 15:00:58.059417 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.059420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.059590 (+ 170us) server_negotiation.cc:407] Connection header received
0619 15:00:58.059668 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.059670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.059699 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.059758 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:58.060353 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.060799 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.061550 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.061740 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.061810 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.062853 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:58.062855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:58.062856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:00:58.062874 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:58.064069 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:58.064377 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:58.064379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:58.064380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:58.064442 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:00:58.064620 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:00:58.064621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:00:58.064622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:00:58.064768 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:58.064853 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.065011 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.065081 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:00:58.067214 27580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:00:58.069918 27580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:00:58.082008 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.076727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:43907 (local address 127.0.0.1:33444)
0619 15:00:58.077029 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.077071 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.077180 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.077985 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.077990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.078018 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:58.078111 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.078116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.079392 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.079395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.080504 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.080511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.081564 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.081567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.081702 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.081739 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.081822 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.081900 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":41,"thread_start_us":79,"threads_started":1}
I20260619 15:00:58.082089 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.077281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.10.71.131:43907)
0619 15:00:58.077559 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.077570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.077591 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:00:58.077651 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.077657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.077830 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.078239 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:58.078336 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.079267 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.080623 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.081422 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.081676 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.081796 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.081846 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.081928 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20260619 15:00:58.082664 28056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.077752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:39651 (local address 127.0.0.1:38088)
0619 15:00:58.078441 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.078452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.078565 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.078899 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.078901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.078916 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:58.079006 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.079013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.080054 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.080057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.081071 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.081080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.082355 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.082358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.082453 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.082475 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.082528 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.082572 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":82,"threads_started":1}
I20260619 15:00:58.082717 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.077076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.10.71.129:39061)
0619 15:00:58.077414 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.077426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.077882 (+ 456us) server_negotiation.cc:407] Connection header received
0619 15:00:58.077958 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.077962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.078122 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.078238 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:58.078594 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.079489 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.080635 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.081423 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.081542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.081676 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.082508 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.082598 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":141,"threads_started":1}
I20260619 15:00:58.083194 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.078110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.10.71.130:39651)
0619 15:00:58.078372 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.078382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.078537 (+ 155us) server_negotiation.cc:407] Connection header received
0619 15:00:58.078658 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.078663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.078787 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.078916 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:58.079087 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.079926 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.081195 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.081987 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.082117 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.082241 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.083006 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.083075 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260619 15:00:58.082671 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.076910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:39061 (local address 127.0.0.1:50368)
0619 15:00:58.077717 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.077728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.077801 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.078376 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.078379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.078394 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:58.078468 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.078474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.079601 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.079606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.080519 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.080528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.081548 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.081552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.082378 (+ 826us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.082398 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.082468 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.082544 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260619 15:00:58.084800 27975 tablet_service.cc:1467] Processing CreateTablet for tablet 67460a8e7d6840c9a9a4dca7587189c1 (DEFAULT_TABLE table=test-table [id=470a6e5dcf7743b7b338b7fde8d83a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:58.085649 27975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67460a8e7d6840c9a9a4dca7587189c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:58.085486 27841 tablet_service.cc:1467] Processing CreateTablet for tablet 67460a8e7d6840c9a9a4dca7587189c1 (DEFAULT_TABLE table=test-table [id=470a6e5dcf7743b7b338b7fde8d83a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:58.086285 27841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67460a8e7d6840c9a9a4dca7587189c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:58.090950 28058 tablet_bootstrap.cc:492] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Bootstrap starting.
I20260619 15:00:58.091344 28059 tablet_bootstrap.cc:492] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Bootstrap starting.
I20260619 15:00:58.093214 28058 tablet_bootstrap.cc:654] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:58.093576 28059 tablet_bootstrap.cc:654] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:58.094174 28058 log.cc:824] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:58.094159 27707 tablet_service.cc:1467] Processing CreateTablet for tablet 67460a8e7d6840c9a9a4dca7587189c1 (DEFAULT_TABLE table=test-table [id=470a6e5dcf7743b7b338b7fde8d83a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:58.094622 28059 log.cc:824] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:58.095016 27707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67460a8e7d6840c9a9a4dca7587189c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:58.095815 28058 tablet_bootstrap.cc:492] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: No bootstrap required, opened a new log
I20260619 15:00:58.096004 28058 ts_tablet_manager.cc:1397] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:00:58.096303 28059 tablet_bootstrap.cc:492] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: No bootstrap required, opened a new log
I20260619 15:00:58.096498 28059 ts_tablet_manager.cc:1397] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260619 15:00:58.099012 28058 raft_consensus.cc:348] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.099210 28058 raft_consensus.cc:374] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:58.099256 28058 raft_consensus.cc:729] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a93046a56440c783a7e6de8141bf4e, State: Initialized, Role: FOLLOWER
I20260619 15:00:58.099643 28058 consensus_queue.cc:260] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.100152 28041 heartbeater.cc:503] Master 127.10.71.190:37387 was elected leader, sending a full tablet report...
I20260619 15:00:58.100085 28059 raft_consensus.cc:348] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.100286 28059 raft_consensus.cc:374] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:58.100353 28059 raft_consensus.cc:729] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95b077931c149f8aeeb57dd49efaddf, State: Initialized, Role: FOLLOWER
I20260619 15:00:58.100479 28058 ts_tablet_manager.cc:1428] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 15:00:58.100756 28059 consensus_queue.cc:260] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.101143 28063 tablet_bootstrap.cc:492] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: Bootstrap starting.
I20260619 15:00:58.102843 28063 tablet_bootstrap.cc:654] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:58.103005 28059 ts_tablet_manager.cc:1428] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260619 15:00:58.103298 27907 heartbeater.cc:503] Master 127.10.71.190:37387 was elected leader, sending a full tablet report...
I20260619 15:00:58.103817 28063 log.cc:824] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:58.105618 28063 tablet_bootstrap.cc:492] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: No bootstrap required, opened a new log
I20260619 15:00:58.105742 28063 ts_tablet_manager.cc:1397] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:00:58.108068 28063 raft_consensus.cc:348] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.108222 28063 raft_consensus.cc:374] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:58.108254 28063 raft_consensus.cc:729] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c690e12c5eb41d3b8d9873d5db08f9c, State: Initialized, Role: FOLLOWER
I20260619 15:00:58.108656 28063 consensus_queue.cc:260] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.109138 27773 heartbeater.cc:503] Master 127.10.71.190:37387 was elected leader, sending a full tablet report...
I20260619 15:00:58.109359 28063 ts_tablet_manager.cc:1428] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260619 15:00:58.140848 27774 tablet.cc:2369] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:00:58.286419 28042 tablet.cc:2369] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:00:58.347658 27908 tablet.cc:2369] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:00:58.363608 28066 raft_consensus.cc:482] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:58.363766 28066 raft_consensus.cc:504] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.364790 28066 leader_election.cc:288] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a93046a56440c783a7e6de8141bf4e (127.10.71.131:43907), e95b077931c149f8aeeb57dd49efaddf (127.10.71.130:39651)
I20260619 15:00:58.368598 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.365091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:48721 (local address 127.10.71.130:39651)
0619 15:00:58.365211 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.365216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.365531 (+ 315us) server_negotiation.cc:407] Connection header received
0619 15:00:58.365651 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.365654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.365701 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.365781 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:58.366046 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.366513 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.367597 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.367993 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.368138 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.368190 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.368394 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.368464 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 15:00:58.368603 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.365061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:57729 (local address 127.10.71.131:43907)
0619 15:00:58.365195 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.365202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.365365 (+ 163us) server_negotiation.cc:407] Connection header received
0619 15:00:58.365477 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.365483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.365550 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.365623 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:58.365915 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.366444 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.367598 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.368010 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.368164 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.368213 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.368433 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.368497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:00:58.368614 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.365002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:39651 (local address 127.10.71.129:48721)
0619 15:00:58.365435 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.365448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.365544 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.365802 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.365804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.365817 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:58.365943 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.365949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.366651 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.366654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.367475 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.367483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.368139 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.368142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.368243 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.368265 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.368378 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.368453 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260619 15:00:58.368621 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.364967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:43907 (local address 127.10.71.129:57729)
0619 15:00:58.365283 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.365295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.365379 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.365649 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.365652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.365669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:58.365808 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.365815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.366576 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.366581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.367466 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.367476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.368177 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.368181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.368277 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.368336 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.368415 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.368479 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":48,"thread_start_us":93,"threads_started":1}
I20260619 15:00:58.369179 27995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67460a8e7d6840c9a9a4dca7587189c1" candidate_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a93046a56440c783a7e6de8141bf4e" is_pre_election: true
I20260619 15:00:58.369180 27861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67460a8e7d6840c9a9a4dca7587189c1" candidate_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95b077931c149f8aeeb57dd49efaddf" is_pre_election: true
I20260619 15:00:58.369457 27861 raft_consensus.cc:2454] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c690e12c5eb41d3b8d9873d5db08f9c in term 0.
I20260619 15:00:58.369457 27995 raft_consensus.cc:2454] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c690e12c5eb41d3b8d9873d5db08f9c in term 0.
I20260619 15:00:58.369894 27661 leader_election.cc:302] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c690e12c5eb41d3b8d9873d5db08f9c, e95b077931c149f8aeeb57dd49efaddf; no voters:
I20260619 15:00:58.370169 28066 raft_consensus.cc:2781] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:00:58.370252 28066 raft_consensus.cc:482] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:00:58.370293 28066 raft_consensus.cc:3037] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:58.371281 28066 raft_consensus.cc:504] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.371591 28066 leader_election.cc:288] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [CANDIDATE]: Term 1 election: Requested vote from peers e1a93046a56440c783a7e6de8141bf4e (127.10.71.131:43907), e95b077931c149f8aeeb57dd49efaddf (127.10.71.130:39651)
I20260619 15:00:58.371865 27995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67460a8e7d6840c9a9a4dca7587189c1" candidate_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a93046a56440c783a7e6de8141bf4e"
I20260619 15:00:58.371923 27861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67460a8e7d6840c9a9a4dca7587189c1" candidate_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95b077931c149f8aeeb57dd49efaddf"
I20260619 15:00:58.371978 27995 raft_consensus.cc:3037] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:58.372031 27861 raft_consensus.cc:3037] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:58.373247 27861 raft_consensus.cc:2454] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c690e12c5eb41d3b8d9873d5db08f9c in term 1.
I20260619 15:00:58.373247 27995 raft_consensus.cc:2454] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c690e12c5eb41d3b8d9873d5db08f9c in term 1.
I20260619 15:00:58.373523 27661 leader_election.cc:302] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c690e12c5eb41d3b8d9873d5db08f9c, e95b077931c149f8aeeb57dd49efaddf; no voters:
I20260619 15:00:58.373700 28066 raft_consensus.cc:2781] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:00:58.373947 28066 raft_consensus.cc:686] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 1 LEADER]: Becoming Leader. State: Replica: 2c690e12c5eb41d3b8d9873d5db08f9c, State: Running, Role: LEADER
I20260619 15:00:58.374240 28066 consensus_queue.cc:237] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } }
I20260619 15:00:58.377110 27580 catalog_manager.cc:5495] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c690e12c5eb41d3b8d9873d5db08f9c (127.10.71.129). New cstate: current_term: 1 leader_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c690e12c5eb41d3b8d9873d5db08f9c" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 39061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 } health_report { overall_health: UNKNOWN } } }
I20260619 15:00:58.438835 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.435676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.10.71.129:39061)
0619 15:00:58.435805 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.435809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.435822 (+ 13us) server_negotiation.cc:407] Connection header received
0619 15:00:58.435961 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.435963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.436014 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.436121 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:00:58.436375 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.436823 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.437596 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.437793 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.437890 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.438172 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:00:58.438497 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:00:58.438619 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.438672 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.438716 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 15:00:58.443974 27995 raft_consensus.cc:1261] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 1 FOLLOWER]: Refusing update from remote peer 2c690e12c5eb41d3b8d9873d5db08f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:00:58.444547 28066 consensus_queue.cc:1035] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a93046a56440c783a7e6de8141bf4e" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:00:58.444598 27861 raft_consensus.cc:1261] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 1 FOLLOWER]: Refusing update from remote peer 2c690e12c5eb41d3b8d9873d5db08f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:00:58.445189 28066 consensus_queue.cc:1035] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95b077931c149f8aeeb57dd49efaddf" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:00:58.454578 28073 mvcc.cc:205] Tried to move back new op lower bound from 7298585634579677184 to 7298585634303410176. Current Snapshot: MvccSnapshot[applied={T|T < 7298585634579677184}]
I20260619 15:00:58.458760 27580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:00:58.458913 27580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:00:58.461033 27580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17ed192e2207425b9ab175d1036d157d: Sending DeleteTablet for 3 replicas of tablet 67460a8e7d6840c9a9a4dca7587189c1
I20260619 15:00:58.461675 27975 tablet_service.cc:1514] Processing DeleteTablet for tablet 67460a8e7d6840c9a9a4dca7587189c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:58 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:33444
I20260619 15:00:58.461755 27707 tablet_service.cc:1514] Processing DeleteTablet for tablet 67460a8e7d6840c9a9a4dca7587189c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:58 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:50368
I20260619 15:00:58.462114 28082 tablet_replica.cc:330] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: stopping tablet replica
I20260619 15:00:58.462184 27841 tablet_service.cc:1514] Processing DeleteTablet for tablet 67460a8e7d6840c9a9a4dca7587189c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:58 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:38088
I20260619 15:00:58.462395 28082 raft_consensus.cc:2229] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:00:58.462582 28084 tablet_replica.cc:330] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: stopping tablet replica
I20260619 15:00:58.462693 28082 raft_consensus.cc:2258] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:58.462860 28084 raft_consensus.cc:2229] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:00:58.462872 28083 tablet_replica.cc:330] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c: stopping tablet replica
I20260619 15:00:58.463109 28083 raft_consensus.cc:2229] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:00:58.463143 28084 raft_consensus.cc:2258] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:58.463187 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27644
I20260619 15:00:58.463411 28083 raft_consensus.cc:2258] T 67460a8e7d6840c9a9a4dca7587189c1 P 2c690e12c5eb41d3b8d9873d5db08f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:58.463976 28082 ts_tablet_manager.cc:1905] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:58.465551 28084 ts_tablet_manager.cc:1905] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:58.466518 28082 ts_tablet_manager.cc:1918] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:58.466610 28082 log.cc:1196] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/67460a8e7d6840c9a9a4dca7587189c1
I20260619 15:00:58.466939 28082 ts_tablet_manager.cc:1939] T 67460a8e7d6840c9a9a4dca7587189c1 P e1a93046a56440c783a7e6de8141bf4e: Deleting consensus metadata
I20260619 15:00:58.468001 28084 ts_tablet_manager.cc:1918] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:58.468022 27566 catalog_manager.cc:4841] TS e1a93046a56440c783a7e6de8141bf4e (127.10.71.131:43907): tablet 67460a8e7d6840c9a9a4dca7587189c1 (table test-table [id=470a6e5dcf7743b7b338b7fde8d83a7d]) successfully deleted
I20260619 15:00:58.468088 28084 log.cc:1196] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/67460a8e7d6840c9a9a4dca7587189c1
I20260619 15:00:58.468417 28084 ts_tablet_manager.cc:1939] T 67460a8e7d6840c9a9a4dca7587189c1 P e95b077931c149f8aeeb57dd49efaddf: Deleting consensus metadata
I20260619 15:00:58.469246 27567 catalog_manager.cc:4841] TS e95b077931c149f8aeeb57dd49efaddf (127.10.71.130:39651): tablet 67460a8e7d6840c9a9a4dca7587189c1 (table test-table [id=470a6e5dcf7743b7b338b7fde8d83a7d]) successfully deleted
W20260619 15:00:58.470320 27568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:39061 (error 108)
I20260619 15:00:58.470363 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27778
I20260619 15:00:58.472501 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.472221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:39061 (local address 127.0.0.1:50378)
0619 15:00:58.472389 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.472443 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 15:00:58.472632 27568 catalog_manager.cc:4584] TS 2c690e12c5eb41d3b8d9873d5db08f9c (127.10.71.129:39061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67460a8e7d6840c9a9a4dca7587189c1: Network error: Client connection negotiation failed: client connection to 127.10.71.129:39061: connect: Connection refused (error 111)
I20260619 15:00:58.475404 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27912
I20260619 15:00:58.480593 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 27550
2026-06-19T15:00:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2369 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T15:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:00:58Z Disabled control of system clock
I20260619 15:00:58.508226 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:41391
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:00:58.606300 28089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:58.606501 28089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:58.606531 28089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:58.609990 28089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:00:58.610030 28089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:58.610044 28089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:58.610056 28089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:00:58.610069 28089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:00:58.614166 28089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:41391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:58.614904 28089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:58.615965 28089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:00:58.623298 28089 server_base.cc:1034] running on GCE node
W20260619 15:00:58.623256 28095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:58.623267 28094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:58.623256 28097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:58.624007 28089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:58.624977 28089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:58.626133 28089 hybrid_clock.cc:648] HybridClock initialized: now 1781881258626106 us; error 36 us; skew 500 ppm
I20260619 15:00:58.627822 28089 webserver.cc:460] Webserver started at http://127.10.71.190:37025/ using document root <none> and password file <none>
I20260619 15:00:58.628393 28089 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:58.628470 28089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:58.628674 28089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:58.630420 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "8623c432a2e3494ca3d0bafadfb7b8a2"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "0f8247a0841f2c28d8cca7228ce6c582"
server_key_iv: "2efb1df4994beec24e4cc135c98c3771"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.630977 28089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "8623c432a2e3494ca3d0bafadfb7b8a2"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "0f8247a0841f2c28d8cca7228ce6c582"
server_key_iv: "2efb1df4994beec24e4cc135c98c3771"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.634249 28089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:00:58.636413 28103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.637408 28089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:00:58.637544 28089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "8623c432a2e3494ca3d0bafadfb7b8a2"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "0f8247a0841f2c28d8cca7228ce6c582"
server_key_iv: "2efb1df4994beec24e4cc135c98c3771"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.637643 28089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:58.660156 28089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:58.660852 28089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:58.661044 28089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:58.667101 28089 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:41391
I20260619 15:00:58.667115 28155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:41391 every 8 connection(s)
I20260619 15:00:58.667963 28089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:00:58.670588 28156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:58.672821 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28089
I20260619 15:00:58.672947 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:00:58.673187 10526 external_mini_cluster.cc:1428] Setting key 25a86d8aae350602f2e68d08a6ccefa8
I20260619 15:00:58.675891 28156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: Bootstrap starting.
I20260619 15:00:58.677913 28156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:58.678557 28156 log.cc:824] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:58.680192 28156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: No bootstrap required, opened a new log
I20260619 15:00:58.681648 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.674476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.10.71.190:41391)
0619 15:00:58.675088 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.675100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.675132 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:00:58.675666 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.675681 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.676083 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.676390 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:58.676784 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.677715 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.678578 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.679062 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.679229 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.680049 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:58.680053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:58.680064 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:00:58.680483 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:58.680534 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:00:58.680553 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:58.680656 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.680767 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.681027 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260619 15:00:58.683547 28156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41391 } }
I20260619 15:00:58.683769 28156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:58.683842 28156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8623c432a2e3494ca3d0bafadfb7b8a2, State: Initialized, Role: FOLLOWER
I20260619 15:00:58.684242 28156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41391 } }
I20260619 15:00:58.684402 28156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:00:58.684458 28156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:00:58.684545 28156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:58.685256 28156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41391 } }
I20260619 15:00:58.685559 28156 leader_election.cc:302] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8623c432a2e3494ca3d0bafadfb7b8a2; no voters:
I20260619 15:00:58.685839 28156 leader_election.cc:288] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:00:58.685935 28161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:00:58.686162 28161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [term 1 LEADER]: Becoming Leader. State: Replica: 8623c432a2e3494ca3d0bafadfb7b8a2, State: Running, Role: LEADER
I20260619 15:00:58.686482 28161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41391 } }
I20260619 15:00:58.686908 28156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:00:58.688073 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8623c432a2e3494ca3d0bafadfb7b8a2. Latest consensus state: current_term: 1 leader_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41391 } } }
I20260619 15:00:58.688184 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [sys.catalog]: This master's current role is: LEADER
I20260619 15:00:58.688432 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8623c432a2e3494ca3d0bafadfb7b8a2" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 41391 } } }
I20260619 15:00:58.688555 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2 [sys.catalog]: This master's current role is: LEADER
W20260619 15:00:58.691526 28176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:00:58.691612 28176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:00:58.691692 28170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:00:58.692387 28170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:00:58.697252 28170 catalog_manager.cc:1347] Generated new cluster ID: b5b95b48f82844eba2019dc6a6fd4284
I20260619 15:00:58.697312 28170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:00:58.710227 28170 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:00:58.711023 28170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:00:58.715699 28170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: Generated new TSK 0
I20260619 15:00:58.716285 28170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:00:58.723206 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41391
--builtin_ntp_servers=127.10.71.148:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:00:58.826743 28180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:58.826936 28180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:58.826967 28180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:58.830495 28180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:58.830534 28180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:58.830578 28180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:00:58.835037 28180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:58.835920 28180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:58.837132 28180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:58.845530 28186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:58.845686 28180 server_base.cc:1034] running on GCE node
W20260619 15:00:58.845580 28185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:58.845808 28188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:58.846174 28180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:58.846660 28180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:58.847805 28180 hybrid_clock.cc:648] HybridClock initialized: now 1781881258847782 us; error 35 us; skew 500 ppm
I20260619 15:00:58.849364 28180 webserver.cc:460] Webserver started at http://127.10.71.129:45803/ using document root <none> and password file <none>
I20260619 15:00:58.849809 28180 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:58.849859 28180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:58.849992 28180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:58.851692 28180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "04645c80e0504aab9372276016acb740"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "c0ab025d08b2000a16918490d09d07a2"
server_key_iv: "33c7ae95832b15468ac66b698bc177b5"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.852164 28180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "04645c80e0504aab9372276016acb740"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "c0ab025d08b2000a16918490d09d07a2"
server_key_iv: "33c7ae95832b15468ac66b698bc177b5"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.855165 28180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:00:58.857221 28194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.858227 28180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 15:00:58.858307 28180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "04645c80e0504aab9372276016acb740"
format_stamp: "Formatted at 2026-06-19 15:00:58 on dist-test-slave-3qdh"
server_key: "c0ab025d08b2000a16918490d09d07a2"
server_key_iv: "33c7ae95832b15468ac66b698bc177b5"
server_key_version: "encryptionkey@0"
I20260619 15:00:58.858378 28180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:58.891029 28180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:58.891562 28180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:58.891697 28180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:58.892156 28180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:58.892999 28180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:58.893043 28180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.893079 28180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:58.893101 28180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:58.900694 28180 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:33855
I20260619 15:00:58.900768 28307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:33855 every 8 connection(s)
I20260619 15:00:58.901587 28180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 15:00:58.908825 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28180
I20260619 15:00:58.908973 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 15:00:58.909284 10526 external_mini_cluster.cc:1428] Setting key ea81287722982a203cbbaebafab72d88
I20260619 15:00:58.909641 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.903136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:35123 (local address 127.10.71.190:41391)
0619 15:00:58.903255 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:58.903258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:58.904152 (+ 894us) server_negotiation.cc:407] Connection header received
0619 15:00:58.905412 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:58.905415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:58.905473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:58.905545 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:58.906307 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.906797 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.907831 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.908054 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.908157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.909201 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:58.909204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:58.909207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:00:58.909220 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:58.909242 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:00:58.909251 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:58.909324 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:58.909464 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 15:00:58.909517 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 15:00:58.910427 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:58.903412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41391 (local address 127.10.71.129:35123)
0619 15:00:58.904017 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:58.904052 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:58.905212 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:58.905691 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:58.905697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:58.905769 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:58.906173 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.906183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.906913 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.906917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.907710 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:58.907719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:58.908190 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:58.908193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:58.908355 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:58.908915 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:00:58.908916 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:58.908970 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:00:58.909048 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:58.909344 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:58.909353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:58.909360 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 15:00:58.909451 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 15:00:58.909651 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260619 15:00:58.911154 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41391
--builtin_ntp_servers=127.10.71.148:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:00:58.911599 28308 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41391
I20260619 15:00:58.911798 28308 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:58.912261 28308 heartbeater.cc:511] Master 127.10.71.190:41391 requested a full tablet report, sending...
I20260619 15:00:58.913728 28120 ts_manager.cc:194] Registered new tserver with Master: 04645c80e0504aab9372276016acb740 (127.10.71.129:33855)
I20260619 15:00:58.914712 28120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.129:35123
W20260619 15:00:59.014593 28311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:59.014853 28311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:59.014923 28311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:59.018692 28311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:59.018760 28311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:59.018855 28311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:00:59.023396 28311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:59.024288 28311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:59.025548 28311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:59.034353 28317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:59.034417 28316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:59.034415 28319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:59.034590 28311 server_base.cc:1034] running on GCE node
I20260619 15:00:59.035092 28311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:59.035640 28311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:59.036789 28311 hybrid_clock.cc:648] HybridClock initialized: now 1781881259036772 us; error 26 us; skew 500 ppm
I20260619 15:00:59.038401 28311 webserver.cc:460] Webserver started at http://127.10.71.130:40371/ using document root <none> and password file <none>
I20260619 15:00:59.038904 28311 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:59.038967 28311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:59.039212 28311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:59.041191 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "690314a45691444a9a9b9ede95d4862c"
format_stamp: "Formatted at 2026-06-19 15:00:59 on dist-test-slave-3qdh"
server_key: "2262f9e60a97885bee8cf021dc5f1160"
server_key_iv: "680f63d2db6f4a8c2bf2e40b523d13db"
server_key_version: "encryptionkey@0"
I20260619 15:00:59.041817 28311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "690314a45691444a9a9b9ede95d4862c"
format_stamp: "Formatted at 2026-06-19 15:00:59 on dist-test-slave-3qdh"
server_key: "2262f9e60a97885bee8cf021dc5f1160"
server_key_iv: "680f63d2db6f4a8c2bf2e40b523d13db"
server_key_version: "encryptionkey@0"
I20260619 15:00:59.045341 28311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 15:00:59.047578 28325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:59.048768 28311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 15:00:59.048905 28311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "690314a45691444a9a9b9ede95d4862c"
format_stamp: "Formatted at 2026-06-19 15:00:59 on dist-test-slave-3qdh"
server_key: "2262f9e60a97885bee8cf021dc5f1160"
server_key_iv: "680f63d2db6f4a8c2bf2e40b523d13db"
server_key_version: "encryptionkey@0"
I20260619 15:00:59.049042 28311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:59.065896 28311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:59.066497 28311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:59.066689 28311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:59.067216 28311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:59.068141 28311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:59.068209 28311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:59.068274 28311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:59.068356 28311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:59.077049 28311 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:33763
I20260619 15:00:59.077070 28438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:33763 every 8 connection(s)
I20260619 15:00:59.077901 28311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:00:59.078387 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28311
I20260619 15:00:59.078519 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 15:00:59.078791 10526 external_mini_cluster.cc:1428] Setting key 0848d3cc20bda271c4a6da0bf6753b4a
I20260619 15:00:59.080869 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41391
--builtin_ntp_servers=127.10.71.148:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:00:59.088406 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.079512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:41433 (local address 127.10.71.190:41391)
0619 15:00:59.079626 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.079631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.080771 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 15:00:59.082320 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.082323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.082370 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.082455 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:59.083519 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.084024 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.085957 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.086185 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.086276 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.087789 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:59.087795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:59.087798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:00:59.087821 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:59.087845 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:00:59.087863 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:59.087929 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.088139 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.088229 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 15:00:59.089543 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.079941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41391 (local address 127.10.71.130:41433)
0619 15:00:59.080641 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.080677 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.082074 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.082624 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.082634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.082740 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:59.083371 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.083391 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.084162 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.084169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.085812 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.085827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.086353 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.086360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.086637 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.087334 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:00:59.087338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:59.087413 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:00:59.087509 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:59.087957 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:59.087969 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:59.087978 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.088124 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.088477 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":154,"threads_started":1}
I20260619 15:00:59.090732 28439 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41391
I20260619 15:00:59.090950 28439 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:59.091400 28439 heartbeater.cc:511] Master 127.10.71.190:41391 requested a full tablet report, sending...
I20260619 15:00:59.092417 28120 ts_manager.cc:194] Registered new tserver with Master: 690314a45691444a9a9b9ede95d4862c (127.10.71.130:33763)
I20260619 15:00:59.092938 28120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.130:41433
W20260619 15:00:59.185889 28442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:00:59.186127 28442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:00:59.186188 28442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:00:59.190595 28442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:00:59.190666 28442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:00:59.190747 28442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:00:59.194985 28442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:00:59.195801 28442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:00:59.197072 28442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:00:59.205533 28447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:59.205626 28442 server_base.cc:1034] running on GCE node
W20260619 15:00:59.205539 28450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:00:59.205818 28448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:00:59.206183 28442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:00:59.206683 28442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:00:59.207876 28442 hybrid_clock.cc:648] HybridClock initialized: now 1781881259207853 us; error 33 us; skew 500 ppm
I20260619 15:00:59.209409 28442 webserver.cc:460] Webserver started at http://127.10.71.131:42573/ using document root <none> and password file <none>
I20260619 15:00:59.209868 28442 fs_manager.cc:359] Metadata directory not provided
I20260619 15:00:59.209945 28442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:00:59.210134 28442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:00:59.211826 28442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "45277b9a833241a0b37511371dd3c677"
format_stamp: "Formatted at 2026-06-19 15:00:59 on dist-test-slave-3qdh"
server_key: "9a8bf418974a507c1d9f0d54345e78ee"
server_key_iv: "98844f64e31743ef151b6a22dbd5407f"
server_key_version: "encryptionkey@0"
I20260619 15:00:59.212409 28442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "45277b9a833241a0b37511371dd3c677"
format_stamp: "Formatted at 2026-06-19 15:00:59 on dist-test-slave-3qdh"
server_key: "9a8bf418974a507c1d9f0d54345e78ee"
server_key_iv: "98844f64e31743ef151b6a22dbd5407f"
server_key_version: "encryptionkey@0"
I20260619 15:00:59.215416 28442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:00:59.217478 28456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:59.218492 28442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:00:59.218617 28442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "45277b9a833241a0b37511371dd3c677"
format_stamp: "Formatted at 2026-06-19 15:00:59 on dist-test-slave-3qdh"
server_key: "9a8bf418974a507c1d9f0d54345e78ee"
server_key_iv: "98844f64e31743ef151b6a22dbd5407f"
server_key_version: "encryptionkey@0"
I20260619 15:00:59.218716 28442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:00:59.227797 28442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:00:59.228410 28442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:00:59.228576 28442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:00:59.229063 28442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:00:59.229856 28442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:00:59.229918 28442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:59.229974 28442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:00:59.230024 28442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:00:59.238080 28442 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:45227
I20260619 15:00:59.238106 28569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:45227 every 8 connection(s)
I20260619 15:00:59.238826 28442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 15:00:59.246461 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28442
I20260619 15:00:59.246573 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 15:00:59.246886 10526 external_mini_cluster.cc:1428] Setting key b0a1de32bd607a5637b5277e1e7452c4
I20260619 15:00:59.246942 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.240381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:41371 (local address 127.10.71.190:41391)
0619 15:00:59.240521 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.240544 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.241572 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 15:00:59.242943 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.242945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.242989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.243059 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:59.243797 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.244282 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.245332 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.245533 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.245603 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.246465 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:59.246468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:59.246471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:00:59.246486 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:59.246507 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:00:59.246518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:59.246589 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.246756 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.246813 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:00:59.247861 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.240767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:41391 (local address 127.10.71.131:41371)
0619 15:00:59.241372 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.241407 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.242759 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.243198 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.243203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.243274 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:00:59.243686 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.243697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.244429 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.244432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.245235 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.245243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.245639 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.245642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.245771 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.246235 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:00:59.246237 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:00:59.246268 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:00:59.246330 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:00:59.246612 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:00:59.246622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:00:59.246629 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.246729 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.246904 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260619 15:00:59.248975 28570 heartbeater.cc:347] Connected to a master server at 127.10.71.190:41391
I20260619 15:00:59.249190 28570 heartbeater.cc:464] Registering TS with master...
I20260619 15:00:59.249621 28570 heartbeater.cc:511] Master 127.10.71.190:41391 requested a full tablet report, sending...
I20260619 15:00:59.250548 28120 ts_manager.cc:194] Registered new tserver with Master: 45277b9a833241a0b37511371dd3c677 (127.10.71.131:45227)
I20260619 15:00:59.251073 28120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.131:41371
I20260619 15:00:59.260373 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:00:59.265316 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.262203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.10.71.190:41391)
0619 15:00:59.262403 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.262406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.262522 (+ 116us) server_negotiation.cc:407] Connection header received
0619 15:00:59.262609 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.262611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.262644 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.262707 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:00:59.262936 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.263387 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.264115 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.264333 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.264398 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.264999 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:00:59.265002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:00:59.265004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:00:59.265013 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:00:59.265028 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:00:59.265035 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:00:59.265108 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.265170 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.265227 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:00:59.267876 28120 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:00:59.270601 28120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:00:59.283978 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.279053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.10.71.129:33855)
0619 15:00:59.279727 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.279737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.279759 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:00:59.279823 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.279827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.279979 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.280140 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:59.280340 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.281146 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.282501 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.283223 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.283329 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.283489 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.283726 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.283817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":88,"threads_started":1}
I20260619 15:00:59.283998 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.278911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33855 (local address 127.0.0.1:51770)
0619 15:00:59.279415 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.279428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.279503 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.280132 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.280135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.280152 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:59.280225 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.280230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.281253 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.281259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.282354 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.282363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.283341 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.283345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.283525 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.283577 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.283683 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.283833 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":145,"thread_start_us":81,"threads_started":1}
I20260619 15:00:59.283996 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.277733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:33763 (local address 127.0.0.1:44866)
0619 15:00:59.278063 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.278117 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.278227 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.278986 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.278992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.279014 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:59.279137 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.279146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.280680 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.280686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.282320 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.282330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.283341 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.283345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.283545 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.283581 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.283683 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.283833 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":250,"thread_start_us":68,"threads_started":1}
I20260619 15:00:59.284891 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.277913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:45227 (local address 127.0.0.1:40996)
0619 15:00:59.278841 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.278855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.278933 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.279558 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.279560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.279574 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:59.279644 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.279648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.280735 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.280737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.282330 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.282336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.284456 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.284460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.284561 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.284582 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.284655 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.284734 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":122,"thread_start_us":97,"threads_started":1}
I20260619 15:00:59.285076 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.277854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.10.71.130:33763)
0619 15:00:59.278490 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.278501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.278524 (+ 23us) server_negotiation.cc:407] Connection header received
0619 15:00:59.278615 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.278620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.278810 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.279315 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:59.279475 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.280558 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.282450 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.283223 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.283329 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.283489 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.284168 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.284933 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":132,"threads_started":1}
I20260619 15:00:59.285317 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.278433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.10.71.131:45227)
0619 15:00:59.278794 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.278804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.279031 (+ 227us) server_negotiation.cc:407] Connection header received
0619 15:00:59.279095 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.279100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.279290 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.279429 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:59.279750 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.280606 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.283551 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.284295 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.284438 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.284585 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.285032 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.285137 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":140,"threads_started":1}
I20260619 15:00:59.286860 28242 tablet_service.cc:1467] Processing CreateTablet for tablet fa8f68dbce6347f685cd24fff61e0139 (DEFAULT_TABLE table=test-table [id=f49eb99b689f467ca491af12c5718e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:59.287458 28504 tablet_service.cc:1467] Processing CreateTablet for tablet fa8f68dbce6347f685cd24fff61e0139 (DEFAULT_TABLE table=test-table [id=f49eb99b689f467ca491af12c5718e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:59.287475 28373 tablet_service.cc:1467] Processing CreateTablet for tablet fa8f68dbce6347f685cd24fff61e0139 (DEFAULT_TABLE table=test-table [id=f49eb99b689f467ca491af12c5718e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:00:59.287906 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8f68dbce6347f685cd24fff61e0139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:59.288450 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8f68dbce6347f685cd24fff61e0139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:59.288486 28504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8f68dbce6347f685cd24fff61e0139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:00:59.294275 28587 tablet_bootstrap.cc:492] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Bootstrap starting.
I20260619 15:00:59.295498 28588 tablet_bootstrap.cc:492] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: Bootstrap starting.
I20260619 15:00:59.296221 28587 tablet_bootstrap.cc:654] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:59.296976 28587 log.cc:824] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:59.297154 28588 tablet_bootstrap.cc:654] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:59.297626 28589 tablet_bootstrap.cc:492] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Bootstrap starting.
I20260619 15:00:59.297926 28588 log.cc:824] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:59.299027 28587 tablet_bootstrap.cc:492] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: No bootstrap required, opened a new log
I20260619 15:00:59.299203 28587 ts_tablet_manager.cc:1397] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:00:59.299602 28589 tablet_bootstrap.cc:654] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Neither blocks nor log segments found. Creating new log.
I20260619 15:00:59.299902 28588 tablet_bootstrap.cc:492] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: No bootstrap required, opened a new log
I20260619 15:00:59.300109 28588 ts_tablet_manager.cc:1397] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:00:59.300369 28589 log.cc:824] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:00:59.302637 28589 tablet_bootstrap.cc:492] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: No bootstrap required, opened a new log
I20260619 15:00:59.302834 28589 ts_tablet_manager.cc:1397] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 15:00:59.303274 28588 raft_consensus.cc:348] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.303481 28588 raft_consensus.cc:374] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:59.303548 28588 raft_consensus.cc:729] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04645c80e0504aab9372276016acb740, State: Initialized, Role: FOLLOWER
I20260619 15:00:59.303558 28587 raft_consensus.cc:348] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.303821 28587 raft_consensus.cc:374] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:59.303897 28587 raft_consensus.cc:729] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45277b9a833241a0b37511371dd3c677, State: Initialized, Role: FOLLOWER
I20260619 15:00:59.303993 28588 consensus_queue.cc:260] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.304309 28587 consensus_queue.cc:260] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.304684 28308 heartbeater.cc:503] Master 127.10.71.190:41391 was elected leader, sending a full tablet report...
I20260619 15:00:59.305009 28588 ts_tablet_manager.cc:1428] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260619 15:00:59.305255 28587 ts_tablet_manager.cc:1428] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 15:00:59.305576 28570 heartbeater.cc:503] Master 127.10.71.190:41391 was elected leader, sending a full tablet report...
I20260619 15:00:59.306465 28589 raft_consensus.cc:348] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.306686 28589 raft_consensus.cc:374] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:00:59.306744 28589 raft_consensus.cc:729] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690314a45691444a9a9b9ede95d4862c, State: Initialized, Role: FOLLOWER
I20260619 15:00:59.307078 28589 consensus_queue.cc:260] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.307703 28439 heartbeater.cc:503] Master 127.10.71.190:41391 was elected leader, sending a full tablet report...
I20260619 15:00:59.307950 28589 ts_tablet_manager.cc:1428] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 15:00:59.328763 28440 tablet.cc:2369] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:00:59.402634 28309 tablet.cc:2369] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:00:59.489672 28571 tablet.cc:2369] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:00:59.521602 28595 raft_consensus.cc:482] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:00:59.521777 28595 raft_consensus.cc:504] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.522938 28595 leader_election.cc:288] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45277b9a833241a0b37511371dd3c677 (127.10.71.131:45227), 04645c80e0504aab9372276016acb740 (127.10.71.129:33855)
I20260619 15:00:59.527438 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.523222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:49099 (local address 127.10.71.131:45227)
0619 15:00:59.523348 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.523352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.523366 (+ 14us) server_negotiation.cc:407] Connection header received
0619 15:00:59.523417 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.523421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.523471 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.523529 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:59.523841 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.524621 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.526195 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.526762 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.526878 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.526926 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.527226 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.527318 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 15:00:59.527438 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.523352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:60781 (local address 127.10.71.129:33855)
0619 15:00:59.523459 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.523463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.523664 (+ 201us) server_negotiation.cc:407] Connection header received
0619 15:00:59.523728 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.523730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.523767 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.523852 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:00:59.524109 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.524839 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.526195 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.526762 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.526878 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.526926 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.527226 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.527318 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 15:00:59.527449 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.523099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:45227 (local address 127.10.71.130:49099)
0619 15:00:59.523214 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.523230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.523308 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.523563 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.523567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.523587 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:59.523724 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.523732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.524725 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.524728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.526076 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.526085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.526878 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.526881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.527080 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.527125 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.527194 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.527311 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":111}
I20260619 15:00:59.527456 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.523184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33855 (local address 127.10.71.130:60781)
0619 15:00:59.523560 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.523571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:00:59.523645 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:00:59.523889 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:00:59.523891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:00:59.523905 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:00:59.523998 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.524002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.525037 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.525040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.526097 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:00:59.526102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.526881 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.526884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:00:59.527084 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.527119 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:00:59.527194 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 15:00:59.527311 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":108,"thread_start_us":118,"threads_started":1}
I20260619 15:00:59.528162 28524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8f68dbce6347f685cd24fff61e0139" candidate_uuid: "690314a45691444a9a9b9ede95d4862c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45277b9a833241a0b37511371dd3c677" is_pre_election: true
I20260619 15:00:59.528240 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8f68dbce6347f685cd24fff61e0139" candidate_uuid: "690314a45691444a9a9b9ede95d4862c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04645c80e0504aab9372276016acb740" is_pre_election: true
I20260619 15:00:59.528460 28262 raft_consensus.cc:2454] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690314a45691444a9a9b9ede95d4862c in term 0.
I20260619 15:00:59.528458 28524 raft_consensus.cc:2454] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690314a45691444a9a9b9ede95d4862c in term 0.
I20260619 15:00:59.528964 28327 leader_election.cc:302] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04645c80e0504aab9372276016acb740, 690314a45691444a9a9b9ede95d4862c; no voters:
I20260619 15:00:59.529218 28595 raft_consensus.cc:2781] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:00:59.529297 28595 raft_consensus.cc:482] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:00:59.529336 28595 raft_consensus.cc:3037] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:59.530285 28595 raft_consensus.cc:504] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.530627 28595 leader_election.cc:288] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [CANDIDATE]: Term 1 election: Requested vote from peers 45277b9a833241a0b37511371dd3c677 (127.10.71.131:45227), 04645c80e0504aab9372276016acb740 (127.10.71.129:33855)
I20260619 15:00:59.530908 28524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8f68dbce6347f685cd24fff61e0139" candidate_uuid: "690314a45691444a9a9b9ede95d4862c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45277b9a833241a0b37511371dd3c677"
I20260619 15:00:59.531028 28262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8f68dbce6347f685cd24fff61e0139" candidate_uuid: "690314a45691444a9a9b9ede95d4862c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04645c80e0504aab9372276016acb740"
I20260619 15:00:59.531051 28524 raft_consensus.cc:3037] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:59.531179 28262 raft_consensus.cc:3037] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:00:59.532056 28524 raft_consensus.cc:2454] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690314a45691444a9a9b9ede95d4862c in term 1.
I20260619 15:00:59.532217 28262 raft_consensus.cc:2454] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690314a45691444a9a9b9ede95d4862c in term 1.
I20260619 15:00:59.532369 28329 leader_election.cc:302] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45277b9a833241a0b37511371dd3c677, 690314a45691444a9a9b9ede95d4862c; no voters:
I20260619 15:00:59.532565 28595 raft_consensus.cc:2781] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:00:59.532725 28595 raft_consensus.cc:686] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 1 LEADER]: Becoming Leader. State: Replica: 690314a45691444a9a9b9ede95d4862c, State: Running, Role: LEADER
I20260619 15:00:59.533001 28595 consensus_queue.cc:237] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } }
I20260619 15:00:59.535738 28120 catalog_manager.cc:5495] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c reported cstate change: term changed from 0 to 1, leader changed from <none> to 690314a45691444a9a9b9ede95d4862c (127.10.71.130). New cstate: current_term: 1 leader_uuid: "690314a45691444a9a9b9ede95d4862c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "690314a45691444a9a9b9ede95d4862c" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 33763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 } health_report { overall_health: UNKNOWN } } }
I20260619 15:00:59.573684 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.570604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.10.71.130:33763)
0619 15:00:59.570716 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 15:00:59.570720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:00:59.570736 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:00:59.570898 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:00:59.570900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:00:59.570949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:00:59.571030 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:00:59.571248 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.571714 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.572522 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:00:59.572721 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:00:59.572795 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:00:59.573032 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:00:59.573395 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:00:59.573474 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 15:00:59.573554 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 15:00:59.573593 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 15:00:59.578764 28524 raft_consensus.cc:1261] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 1 FOLLOWER]: Refusing update from remote peer 690314a45691444a9a9b9ede95d4862c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:00:59.578742 28262 raft_consensus.cc:1261] T fa8f68dbce6347f685cd24fff61e0139 P 04645c80e0504aab9372276016acb740 [term 1 FOLLOWER]: Refusing update from remote peer 690314a45691444a9a9b9ede95d4862c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:00:59.579406 28597 consensus_queue.cc:1035] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45277b9a833241a0b37511371dd3c677" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:00:59.579577 28595 consensus_queue.cc:1035] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04645c80e0504aab9372276016acb740" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:00:59.586687 28602 mvcc.cc:205] Tried to move back new op lower bound from 7298585639228178432 to 7298585639049347072. Current Snapshot: MvccSnapshot[applied={T|T < 7298585639228178432}]
I20260619 15:00:59.593772 28113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:00:59.593916 28113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:00:59.596055 28113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8623c432a2e3494ca3d0bafadfb7b8a2: Sending DeleteTablet for 3 replicas of tablet fa8f68dbce6347f685cd24fff61e0139
I20260619 15:00:59.596776 28504 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8f68dbce6347f685cd24fff61e0139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:59 UTC) from {username='slave'} at 127.0.0.1:40996
I20260619 15:00:59.596907 28373 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8f68dbce6347f685cd24fff61e0139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:59 UTC) from {username='slave'} at 127.0.0.1:44866
I20260619 15:00:59.597379 28611 tablet_replica.cc:330] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: stopping tablet replica
I20260619 15:00:59.597599 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8f68dbce6347f685cd24fff61e0139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:00:59 UTC) from {username='slave'} at 127.0.0.1:51770
I20260619 15:00:59.597690 28611 raft_consensus.cc:2229] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:00:59.597704 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28180
I20260619 15:00:59.597967 28611 raft_consensus.cc:2258] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:59.598318 28610 tablet_replica.cc:330] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: stopping tablet replica
I20260619 15:00:59.598636 28610 raft_consensus.cc:2229] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:00:59.598932 28610 raft_consensus.cc:2258] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:00:59.599574 28611 ts_tablet_manager.cc:1905] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:59.600184 28610 ts_tablet_manager.cc:1905] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:00:59.602622 28611 ts_tablet_manager.cc:1918] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:59.602794 28611 log.cc:1196] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/fa8f68dbce6347f685cd24fff61e0139
I20260619 15:00:59.603044 28610 ts_tablet_manager.cc:1918] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:00:59.603209 28610 log.cc:1196] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/fa8f68dbce6347f685cd24fff61e0139
I20260619 15:00:59.603250 28611 ts_tablet_manager.cc:1939] T fa8f68dbce6347f685cd24fff61e0139 P 690314a45691444a9a9b9ede95d4862c: Deleting consensus metadata
I20260619 15:00:59.603596 28610 ts_tablet_manager.cc:1939] T fa8f68dbce6347f685cd24fff61e0139 P 45277b9a833241a0b37511371dd3c677: Deleting consensus metadata
I20260619 15:00:59.604472 28105 catalog_manager.cc:4841] TS 690314a45691444a9a9b9ede95d4862c (127.10.71.130:33763): tablet fa8f68dbce6347f685cd24fff61e0139 (table test-table [id=f49eb99b689f467ca491af12c5718e7d]) successfully deleted
I20260619 15:00:59.604678 28107 catalog_manager.cc:4841] TS 45277b9a833241a0b37511371dd3c677 (127.10.71.131:45227): tablet fa8f68dbce6347f685cd24fff61e0139 (table test-table [id=f49eb99b689f467ca491af12c5718e7d]) successfully deleted
W20260619 15:00:59.606779 28105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:33855 (error 108)
I20260619 15:00:59.606987 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28311
I20260619 15:00:59.608786 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:00:59.608519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:33855 (local address 127.0.0.1:51774)
0619 15:00:59.608661 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 15:00:59.608723 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 15:00:59.609001 28105 catalog_manager.cc:4584] TS 04645c80e0504aab9372276016acb740 (127.10.71.129:33855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa8f68dbce6347f685cd24fff61e0139: Network error: Client connection negotiation failed: client connection to 127.10.71.129:33855: connect: Connection refused (error 111)
I20260619 15:00:59.612368 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28442
I20260619 15:00:59.617218 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28089
2026-06-19T15:00:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1135 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:00:59 dist-test-slave-3qdh krb5kdc[28615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:00:59 dist-test-slave-3qdh krb5kdc[28615](info): set up 2 sockets
Jun 19 15:00:59 dist-test-slave-3qdh krb5kdc[28615](info): commencing operation
krb5kdc: starting...
W20260619 15:01:00.599747 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:00 dist-test-slave-3qdh krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:00.706372 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34071
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:34071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:00.803086 28628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:00.803316 28628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:00.803380 28628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:00.806823 28628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:00.806890 28628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:00.806921 28628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:00.806979 28628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:00.807009 28628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:00.813182 28628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:34071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:00.814277 28628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:00.815451 28628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:01:00.824038 28628 server_base.cc:1034] running on GCE node
W20260619 15:01:00.824018 28634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:00.824030 28633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:00.824083 28636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:00.824831 28628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:00.825733 28628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:00.826957 28628 hybrid_clock.cc:648] HybridClock initialized: now 1781881260826912 us; error 30 us; skew 500 ppm
Jun 19 15:01:00 dist-test-slave-3qdh krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881260, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:00.829478 28628 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:00.830329 28628 webserver.cc:460] Webserver started at http://127.10.71.190:37771/ using document root <none> and password file <none>
I20260619 15:01:00.830850 28628 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:00.830906 28628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:00.831135 28628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:00.833012 28628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "a24db191cf634f27a1b2829a0166f69d"
format_stamp: "Formatted at 2026-06-19 15:01:00 on dist-test-slave-3qdh"
server_key: "50011ba2de11fe2bc34c0afd6af958b2"
server_key_iv: "d868890dbdc4527dfbd2f23cb1351ebe"
server_key_version: "encryptionkey@0"
I20260619 15:01:00.833542 28628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "a24db191cf634f27a1b2829a0166f69d"
format_stamp: "Formatted at 2026-06-19 15:01:00 on dist-test-slave-3qdh"
server_key: "50011ba2de11fe2bc34c0afd6af958b2"
server_key_iv: "d868890dbdc4527dfbd2f23cb1351ebe"
server_key_version: "encryptionkey@0"
I20260619 15:01:00.836779 28628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:00.838943 28643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:00.839923 28628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:00.840062 28628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "a24db191cf634f27a1b2829a0166f69d"
format_stamp: "Formatted at 2026-06-19 15:01:00 on dist-test-slave-3qdh"
server_key: "50011ba2de11fe2bc34c0afd6af958b2"
server_key_iv: "d868890dbdc4527dfbd2f23cb1351ebe"
server_key_version: "encryptionkey@0"
I20260619 15:01:00.840185 28628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:00.850885 28628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:00.853331 28628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:00.853498 28628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:00.859723 28628 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:34071
I20260619 15:01:00.859745 28695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:34071 every 8 connection(s)
I20260619 15:01:00.860571 28628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:00.860862 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28628
I20260619 15:01:00.861019 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:00.861304 10526 external_mini_cluster.cc:1428] Setting key 7a2b3188f43bd401e96620d740d37298
I20260619 15:01:00.863180 28696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:00.867950 28696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d: Bootstrap starting.
Jun 19 15:01:00 dist-test-slave-3qdh krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:00.870281 28696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:00.870919 28696 log.cc:824] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:00.872928 28696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d: No bootstrap required, opened a new log
I20260619 15:01:00.874706 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:00.862715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.10.71.190:34071)
0619 15:01:00.863084 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:00.863098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:00.863127 (+ 29us) server_negotiation.cc:407] Connection header received
0619 15:01:00.863713 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:00.863728 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:00.863997 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:00.864301 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:00.865070 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:00.865941 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:00.866764 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:00.867200 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:00.867353 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:00.870021 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:00.870025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:00.870036 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:00.870065 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:00.871940 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:00.872427 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:00.872432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:00.872437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:00.872572 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:00.872830 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:00.872833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:00.872835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:00.873272 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:00.873451 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:00.873637 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 15:01:00.873889 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":123,"threads_started":1}
I20260619 15:01:00.876834 28696 raft_consensus.cc:348] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24db191cf634f27a1b2829a0166f69d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34071 } }
I20260619 15:01:00.877022 28696 raft_consensus.cc:374] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:00.877056 28696 raft_consensus.cc:729] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24db191cf634f27a1b2829a0166f69d, State: Initialized, Role: FOLLOWER
I20260619 15:01:00.877404 28696 consensus_queue.cc:260] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24db191cf634f27a1b2829a0166f69d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34071 } }
I20260619 15:01:00.877513 28696 raft_consensus.cc:388] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:00.877556 28696 raft_consensus.cc:482] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:00.877620 28696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:00.878391 28696 raft_consensus.cc:504] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24db191cf634f27a1b2829a0166f69d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34071 } }
I20260619 15:01:00.878656 28696 leader_election.cc:302] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a24db191cf634f27a1b2829a0166f69d; no voters:
I20260619 15:01:00.878868 28696 leader_election.cc:288] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:00.878997 28701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:00.879209 28701 raft_consensus.cc:686] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [term 1 LEADER]: Becoming Leader. State: Replica: a24db191cf634f27a1b2829a0166f69d, State: Running, Role: LEADER
I20260619 15:01:00.879570 28701 consensus_queue.cc:237] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24db191cf634f27a1b2829a0166f69d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34071 } }
I20260619 15:01:00.879966 28696 sys_catalog.cc:564] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:00.881253 28703 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a24db191cf634f27a1b2829a0166f69d. Latest consensus state: current_term: 1 leader_uuid: "a24db191cf634f27a1b2829a0166f69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24db191cf634f27a1b2829a0166f69d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34071 } } }
I20260619 15:01:00.881260 28702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a24db191cf634f27a1b2829a0166f69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24db191cf634f27a1b2829a0166f69d" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 34071 } } }
I20260619 15:01:00.881448 28702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:00.881750 28703 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:00.882264 28711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:00.885015 28711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:00.889792 28711 catalog_manager.cc:1347] Generated new cluster ID: 71f3ce3e064c4f218a1f31b7ff5e0d22
I20260619 15:01:00.889848 28711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:00.895828 28711 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:00.896648 28711 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:00.900795 28711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d: Generated new TSK 0
I20260619 15:01:00.901314 28711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 15:01:00.926934 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34071
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:01.027171 28722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:01.027369 28722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:01.027405 28722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:01.031103 28722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:01.031149 28722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:01.031204 28722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:01:01.035867 28722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:01.036799 28722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:01.037669 28722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:01.046092 28727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.046200 28722 server_base.cc:1034] running on GCE node
W20260619 15:01:01.046141 28728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:01.046124 28730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.046703 28722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:01.047245 28722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:01.048370 28722 hybrid_clock.cc:648] HybridClock initialized: now 1781881261048336 us; error 42 us; skew 500 ppm
Jun 19 15:01:01 dist-test-slave-3qdh krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:01.050891 28722 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 15:01:01.051677 28722 webserver.cc:460] Webserver started at http://127.10.71.129:35465/ using document root <none> and password file <none>
I20260619 15:01:01.052201 28722 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:01.052284 28722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:01.052505 28722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:01.054231 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "669d20c78f004efbb907c593c97e6964"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "bd52e62e5985421e5644e17de17b166f"
server_key_iv: "b153d3d2cba35dbce458a8aeb57f04cd"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.054733 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "669d20c78f004efbb907c593c97e6964"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "bd52e62e5985421e5644e17de17b166f"
server_key_iv: "b153d3d2cba35dbce458a8aeb57f04cd"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.057945 28722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 15:01:01.060001 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.061067 28722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:01.061196 28722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "669d20c78f004efbb907c593c97e6964"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "bd52e62e5985421e5644e17de17b166f"
server_key_iv: "b153d3d2cba35dbce458a8aeb57f04cd"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.061300 28722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:01.080850 28722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:01.083756 28722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:01.083932 28722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:01.084426 28722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:01.085084 28722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:01.085131 28722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.085191 28722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:01.085220 28722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.092864 28722 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:37175
I20260619 15:01:01.092886 28850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:37175 every 8 connection(s)
I20260619 15:01:01.093648 28722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
Jun 19 15:01:01 dist-test-slave-3qdh krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:01.101959 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28722
I20260619 15:01:01.102098 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 15:01:01.102430 10526 external_mini_cluster.cc:1428] Setting key 9778cc0473af68347c6ecb57cb513c45
I20260619 15:01:01.106029 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.095189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:40703 (local address 127.10.71.190:34071)
0619 15:01:01.095304 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.095307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.095953 (+ 646us) server_negotiation.cc:407] Connection header received
0619 15:01:01.097160 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.097163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.097219 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.097315 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:01.098265 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.098763 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.099733 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.099948 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.100045 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.102372 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:01.102374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:01.102376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:01.102397 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:01.103838 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.104377 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.104379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.104381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.104446 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.104867 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.104870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.104872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.105059 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:01.105154 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.105709 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.105841 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 15:01:01.106487 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.095426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34071 (local address 127.10.71.129:40703)
0619 15:01:01.095811 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.095844 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.096957 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.097462 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.097468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.097747 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:01.098133 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.098144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.098893 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.098902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.099611 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.099619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.100084 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.100088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.100222 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.100679 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:01.100693 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:01.102227 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:01.103944 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:01.103949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:01.103960 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:01.104260 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:01.104567 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:01.104573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:01.104577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:01.104745 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:01.105160 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:01.105168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:01.105453 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.105618 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.105812 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260619 15:01:01.107506 28851 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34071
I20260619 15:01:01.107735 28851 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:01.108248 28851 heartbeater.cc:511] Master 127.10.71.190:34071 requested a full tablet report, sending...
I20260619 15:01:01.109668 28660 ts_manager.cc:194] Registered new tserver with Master: 669d20c78f004efbb907c593c97e6964 (127.10.71.129:37175)
I20260619 15:01:01.110631 28660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:40703
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 15:01:01.126653 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34071
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:01.225069 28856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:01.225272 28856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:01.225311 28856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:01.228994 28856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:01.229043 28856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:01.229096 28856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:01:01.234019 28856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:01.234798 28856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:01.235607 28856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:01.244479 28861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.244668 28856 server_base.cc:1034] running on GCE node
W20260619 15:01:01.244597 28862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:01.244504 28864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.245203 28856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:01.245683 28856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:01.246827 28856 hybrid_clock.cc:648] HybridClock initialized: now 1781881261246817 us; error 26 us; skew 500 ppm
Jun 19 15:01:01 dist-test-slave-3qdh krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:01.255544 28856 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 15:01:01.256280 28856 webserver.cc:460] Webserver started at http://127.10.71.130:42387/ using document root <none> and password file <none>
I20260619 15:01:01.256819 28856 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:01.256873 28856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:01.257103 28856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:01.258914 28856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "7200069c89b54b738dd32456dc1c8743"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "7a64c82e762eb798d7e6973950e3ddd3"
server_key_iv: "50f24b1c7464b7a904c1cee7bdca5044"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.259431 28856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "7200069c89b54b738dd32456dc1c8743"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "7a64c82e762eb798d7e6973950e3ddd3"
server_key_iv: "50f24b1c7464b7a904c1cee7bdca5044"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.262825 28856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260619 15:01:01.265262 28871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.266346 28856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 15:01:01.266474 28856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "7200069c89b54b738dd32456dc1c8743"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "7a64c82e762eb798d7e6973950e3ddd3"
server_key_iv: "50f24b1c7464b7a904c1cee7bdca5044"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.266574 28856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:01.280016 28856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:01.283031 28856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:01.283171 28856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:01.283587 28856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:01.284260 28856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:01.284349 28856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.284407 28856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:01.284423 28856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.292153 28856 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:37095
I20260619 15:01:01.292222 28984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:37095 every 8 connection(s)
I20260619 15:01:01.292982 28856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
Jun 19 15:01:01 dist-test-slave-3qdh krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:01.302386 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28856
I20260619 15:01:01.302523 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 15:01:01.302794 10526 external_mini_cluster.cc:1428] Setting key 504ee2045c049db2fdccbd137ac9f7f9
I20260619 15:01:01.305006 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.294485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:55183 (local address 127.10.71.190:34071)
0619 15:01:01.294610 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.294614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.295364 (+ 750us) server_negotiation.cc:407] Connection header received
0619 15:01:01.296474 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.296477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.296529 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.296602 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:01.297542 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.298023 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.298991 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.299183 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.299249 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.301339 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:01.301341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:01.301343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:01.301365 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:01.302851 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.303529 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.303531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.303532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.303620 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.303955 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.303957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.303958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.304171 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:01.304276 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.304757 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.304861 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 15:01:01.305672 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.294836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34071 (local address 127.10.71.130:55183)
0619 15:01:01.295234 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.295271 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.296278 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.296736 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.296742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.297044 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:01.297431 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.297443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.298116 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.298118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.298904 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.298912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.299290 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.299292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.299405 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.299884 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:01.299898 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:01.301196 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:01.302976 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:01.302981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:01.302994 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:01.303293 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:01.303707 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:01.303710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:01.303712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:01.303865 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:01.304274 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:01.304280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:01.304514 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.304717 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.304969 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260619 15:01:01.306686 28985 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34071
I20260619 15:01:01.306903 28985 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:01.307372 28985 heartbeater.cc:511] Master 127.10.71.190:34071 requested a full tablet report, sending...
I20260619 15:01:01.308427 28660 ts_manager.cc:194] Registered new tserver with Master: 7200069c89b54b738dd32456dc1c8743 (127.10.71.130:37095)
I20260619 15:01:01.308947 28660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:55183
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 15:01:01.326936 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34071
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:01.425781 28990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:01.426007 28990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:01.426101 28990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:01.429723 28990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:01.429790 28990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:01.429863 28990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:01:01.434610 28990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:01.435456 28990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:01.436331 28990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:01.445024 28995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:01.445003 28996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.445190 28990 server_base.cc:1034] running on GCE node
W20260619 15:01:01.445020 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.445669 28990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:01.446187 28990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:01.447357 28990 hybrid_clock.cc:648] HybridClock initialized: now 1781881261447335 us; error 38 us; skew 500 ppm
Jun 19 15:01:01 dist-test-slave-3qdh krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:01.449723 28990 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 15:01:01.450431 28990 webserver.cc:460] Webserver started at http://127.10.71.131:41009/ using document root <none> and password file <none>
I20260619 15:01:01.450879 28990 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:01.450955 28990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:01.451179 28990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:01.452996 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "6106479de57e468193ec89ba032d4ad2"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "d53c0c8507e5db0e910ee344ff528118"
server_key_iv: "e02a979a3f31a4ff346a56f4769f77f0"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.453491 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "6106479de57e468193ec89ba032d4ad2"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "d53c0c8507e5db0e910ee344ff528118"
server_key_iv: "e02a979a3f31a4ff346a56f4769f77f0"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.456552 28990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:01.458604 29005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.459627 28990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:01.459759 28990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "6106479de57e468193ec89ba032d4ad2"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "d53c0c8507e5db0e910ee344ff528118"
server_key_iv: "e02a979a3f31a4ff346a56f4769f77f0"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.459851 28990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:01.473387 28990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:01.475781 28990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:01.475948 28990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:01.476437 28990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:01.477108 28990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:01.477149 28990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.477208 28990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:01.477259 28990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.485451 28990 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:44625
I20260619 15:01:01.485548 29118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:44625 every 8 connection(s)
I20260619 15:01:01.486186 28990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 15:01:01.491968 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 28990
I20260619 15:01:01.492100 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 15:01:01.492414 10526 external_mini_cluster.cc:1428] Setting key ff1626af2dcff124bb24c96ed578ab32
Jun 19 15:01:01 dist-test-slave-3qdh krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:01.498664 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.487613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:54391 (local address 127.10.71.190:34071)
0619 15:01:01.487738 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.487742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.488453 (+ 711us) server_negotiation.cc:407] Connection header received
0619 15:01:01.489606 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.489608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.489664 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.489760 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:01.490742 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.491289 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.492752 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.493069 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.493192 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.495508 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:01.495511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:01.495513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:01.495536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:01.496831 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.497283 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.497288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.497289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.497362 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.497686 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.497691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.497693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.497858 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:01.497958 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.498412 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.498516 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 15:01:01.499256 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.487910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:34071 (local address 127.10.71.131:54391)
0619 15:01:01.488284 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.488374 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.489409 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.489908 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.489913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.490209 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:01.490607 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.490618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.491413 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.491416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.492636 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.492646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.493244 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.493251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.493415 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.493998 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:01.494012 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:01.495369 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:01.496952 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:01.496957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:01.496968 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:01.497189 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:01.497445 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:01.497449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:01.497451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:01.497595 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:01.497956 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:01.497962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:01.498198 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.498373 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.498635 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260619 15:01:01.500160 29119 heartbeater.cc:347] Connected to a master server at 127.10.71.190:34071
I20260619 15:01:01.500382 29119 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:01.500761 29119 heartbeater.cc:511] Master 127.10.71.190:34071 requested a full tablet report, sending...
I20260619 15:01:01.501807 28660 ts_manager.cc:194] Registered new tserver with Master: 6106479de57e468193ec89ba032d4ad2 (127.10.71.131:44625)
I20260619 15:01:01.502426 28660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:54391
I20260619 15:01:01.505779 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:01:01.514161 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.507576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.10.71.190:34071)
0619 15:01:01.507669 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.507672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.507891 (+ 219us) server_negotiation.cc:407] Connection header received
0619 15:01:01.507986 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.507987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.508016 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.508079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:01.508747 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.509170 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.509886 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.510055 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.510114 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.511364 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:01.511367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:01.511369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:01.511388 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:01.512707 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.513178 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.513180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.513181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.513237 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:01.513471 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:01.513473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:01.513474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:01.513610 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:01.513703 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.513969 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.514071 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 15:01:01.516378 28660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:01:01.518991 28660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:01:01.532163 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.526852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.10.71.131:44625)
0619 15:01:01.527175 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.527186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.527207 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:01:01.527952 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.527957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.528136 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.528244 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:01.528582 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.529221 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.530936 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.531502 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.531626 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.531754 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.531925 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.532022 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
I20260619 15:01:01.532152 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.527063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37175 (local address 127.0.0.1:36502)
0619 15:01:01.527334 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.527347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.527425 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.528109 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.528112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.528128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:01.528222 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.528228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.529475 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.529480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.530613 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.530621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.531704 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.531708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.531807 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.531830 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.531875 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.532032 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":99,"thread_start_us":69,"threads_started":1}
I20260619 15:01:01.532562 29131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.526265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:44625 (local address 127.0.0.1:35898)
0619 15:01:01.526964 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.527019 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.527794 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.528413 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.528415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.528428 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:01.528483 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.528487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.529337 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.529342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.530785 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.530795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.531627 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.531631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.531734 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.531769 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.531899 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.532458 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":207,"thread_start_us":79,"threads_started":1}
I20260619 15:01:01.532645 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.527408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.10.71.129:37175)
0619 15:01:01.527664 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.527674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.527709 (+ 35us) server_negotiation.cc:407] Connection header received
0619 15:01:01.527787 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.527792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.527988 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.528135 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:01.528344 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.529347 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.530729 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.531561 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.532059 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.532192 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.532408 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.532500 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260619 15:01:01.532687 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.526555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.10.71.130:37095)
0619 15:01:01.526830 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.526841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.527426 (+ 585us) server_negotiation.cc:407] Connection header received
0619 15:01:01.527518 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.527524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.527736 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.527902 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:01.528192 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.529291 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.530918 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.531722 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.531807 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.531935 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.532474 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.532570 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
I20260619 15:01:01.532931 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.526401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37095 (local address 127.0.0.1:37404)
0619 15:01:01.527174 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.527186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.527294 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.527901 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.527906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.527928 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:01.528072 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.528081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.529398 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.529401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.530785 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.530795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.531996 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.532002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.532096 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.532116 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.532181 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.532814 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":68,"thread_start_us":79,"threads_started":1}
I20260619 15:01:01.534912 28785 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd44cbae30144d98b9c0ef8f2afcea3 (DEFAULT_TABLE table=test-table [id=b380dbb8e5be48fcbfd3d7aa00a3bb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:01.535095 29053 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd44cbae30144d98b9c0ef8f2afcea3 (DEFAULT_TABLE table=test-table [id=b380dbb8e5be48fcbfd3d7aa00a3bb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:01.535096 28919 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd44cbae30144d98b9c0ef8f2afcea3 (DEFAULT_TABLE table=test-table [id=b380dbb8e5be48fcbfd3d7aa00a3bb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:01.535772 28785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd44cbae30144d98b9c0ef8f2afcea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:01.535913 29053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd44cbae30144d98b9c0ef8f2afcea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:01.535913 28919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd44cbae30144d98b9c0ef8f2afcea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:01.540865 29136 tablet_bootstrap.cc:492] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: Bootstrap starting.
I20260619 15:01:01.542084 29137 tablet_bootstrap.cc:492] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Bootstrap starting.
I20260619 15:01:01.542500 29138 tablet_bootstrap.cc:492] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Bootstrap starting.
I20260619 15:01:01.543776 29136 tablet_bootstrap.cc:654] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:01.543834 29137 tablet_bootstrap.cc:654] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:01.544111 29138 tablet_bootstrap.cc:654] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:01.545024 29137 log.cc:824] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:01.545024 29136 log.cc:824] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:01.545118 29138 log.cc:824] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:01.546550 29136 tablet_bootstrap.cc:492] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: No bootstrap required, opened a new log
I20260619 15:01:01.546552 29137 tablet_bootstrap.cc:492] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: No bootstrap required, opened a new log
I20260619 15:01:01.546716 29137 ts_tablet_manager.cc:1397] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:01:01.546728 29136 ts_tablet_manager.cc:1397] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 15:01:01.549325 29138 tablet_bootstrap.cc:492] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: No bootstrap required, opened a new log
I20260619 15:01:01.549474 29138 ts_tablet_manager.cc:1397] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 15:01:01.549623 29136 raft_consensus.cc:348] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.549814 29136 raft_consensus.cc:374] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:01.549860 29136 raft_consensus.cc:729] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669d20c78f004efbb907c593c97e6964, State: Initialized, Role: FOLLOWER
I20260619 15:01:01.549969 29137 raft_consensus.cc:348] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.550173 29137 raft_consensus.cc:374] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:01.550226 29137 raft_consensus.cc:729] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7200069c89b54b738dd32456dc1c8743, State: Initialized, Role: FOLLOWER
I20260619 15:01:01.550264 29136 consensus_queue.cc:260] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.550594 29137 consensus_queue.cc:260] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.550935 28851 heartbeater.cc:503] Master 127.10.71.190:34071 was elected leader, sending a full tablet report...
I20260619 15:01:01.551245 29136 ts_tablet_manager.cc:1428] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 15:01:01.551445 29137 ts_tablet_manager.cc:1428] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 15:01:01.551751 28985 heartbeater.cc:503] Master 127.10.71.190:34071 was elected leader, sending a full tablet report...
I20260619 15:01:01.552853 29138 raft_consensus.cc:348] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.553087 29138 raft_consensus.cc:374] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:01.553138 29138 raft_consensus.cc:729] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6106479de57e468193ec89ba032d4ad2, State: Initialized, Role: FOLLOWER
I20260619 15:01:01.553455 29138 consensus_queue.cc:260] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.554098 29119 heartbeater.cc:503] Master 127.10.71.190:34071 was elected leader, sending a full tablet report...
I20260619 15:01:01.554306 29138 ts_tablet_manager.cc:1428] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 15:01:01.592949 29144 raft_consensus.cc:482] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:01:01.593123 29144 raft_consensus.cc:504] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.593966 29144 leader_election.cc:288] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7200069c89b54b738dd32456dc1c8743 (127.10.71.130:37095), 669d20c78f004efbb907c593c97e6964 (127.10.71.129:37175)
W20260619 15:01:01.594655 28852 tablet.cc:2369] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:01.598356 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.594495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:44549 (local address 127.10.71.129:37175)
0619 15:01:01.594711 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.594717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.594748 (+ 31us) server_negotiation.cc:407] Connection header received
0619 15:01:01.594863 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.594866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.594910 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.594953 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:01.595299 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.595889 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.597150 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.597688 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.597810 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.597853 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.598196 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.598255 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 15:01:01.598356 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.594393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:53281 (local address 127.10.71.130:37095)
0619 15:01:01.594497 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.594501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.594514 (+ 13us) server_negotiation.cc:407] Connection header received
0619 15:01:01.594559 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.594561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.594606 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.594682 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:01.595032 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.595742 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.597150 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.597688 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.597810 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.597853 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.598196 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.598255 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:01:01.598379 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.594179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:37095 (local address 127.10.71.131:53281)
0619 15:01:01.594285 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.594299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.594372 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.594805 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.594809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.594823 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:01.594921 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.594928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.595832 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.595834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.597033 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.597042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.597817 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.597820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.598029 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.598096 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.598166 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.598249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":98}
I20260619 15:01:01.598394 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.594408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37175 (local address 127.10.71.131:44549)
0619 15:01:01.594675 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.594698 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:01.594757 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:01.595081 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:01.595083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:01.595095 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:01.595191 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.595194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.595964 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.595966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.597052 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:01.597057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.597817 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.597820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:01.598052 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.598101 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 15:01:01.598166 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 15:01:01.598249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":175,"thread_start_us":101,"threads_started":1}
I20260619 15:01:01.598969 28805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd44cbae30144d98b9c0ef8f2afcea3" candidate_uuid: "6106479de57e468193ec89ba032d4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669d20c78f004efbb907c593c97e6964" is_pre_election: true
I20260619 15:01:01.598991 28939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd44cbae30144d98b9c0ef8f2afcea3" candidate_uuid: "6106479de57e468193ec89ba032d4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7200069c89b54b738dd32456dc1c8743" is_pre_election: true
I20260619 15:01:01.599200 28805 raft_consensus.cc:2454] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6106479de57e468193ec89ba032d4ad2 in term 0.
I20260619 15:01:01.599213 28939 raft_consensus.cc:2454] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6106479de57e468193ec89ba032d4ad2 in term 0.
I20260619 15:01:01.599610 29008 leader_election.cc:302] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6106479de57e468193ec89ba032d4ad2, 669d20c78f004efbb907c593c97e6964; no voters:
I20260619 15:01:01.599807 29144 raft_consensus.cc:2781] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:01:01.599879 29144 raft_consensus.cc:482] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:01:01.599917 29144 raft_consensus.cc:3037] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:01.601015 29144 raft_consensus.cc:504] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.601400 29144 leader_election.cc:288] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 7200069c89b54b738dd32456dc1c8743 (127.10.71.130:37095), 669d20c78f004efbb907c593c97e6964 (127.10.71.129:37175)
I20260619 15:01:01.601701 28939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd44cbae30144d98b9c0ef8f2afcea3" candidate_uuid: "6106479de57e468193ec89ba032d4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7200069c89b54b738dd32456dc1c8743"
I20260619 15:01:01.601729 28805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd44cbae30144d98b9c0ef8f2afcea3" candidate_uuid: "6106479de57e468193ec89ba032d4ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669d20c78f004efbb907c593c97e6964"
I20260619 15:01:01.601830 28939 raft_consensus.cc:3037] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:01.601847 28805 raft_consensus.cc:3037] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:01.602757 28805 raft_consensus.cc:2454] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6106479de57e468193ec89ba032d4ad2 in term 1.
I20260619 15:01:01.602761 28939 raft_consensus.cc:2454] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6106479de57e468193ec89ba032d4ad2 in term 1.
I20260619 15:01:01.603027 29008 leader_election.cc:302] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6106479de57e468193ec89ba032d4ad2, 669d20c78f004efbb907c593c97e6964; no voters:
I20260619 15:01:01.603163 29144 raft_consensus.cc:2781] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:01.603341 29144 raft_consensus.cc:686] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 6106479de57e468193ec89ba032d4ad2, State: Running, Role: LEADER
I20260619 15:01:01.603624 29144 consensus_queue.cc:237] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } }
I20260619 15:01:01.606220 28659 catalog_manager.cc:5495] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6106479de57e468193ec89ba032d4ad2 (127.10.71.131). New cstate: current_term: 1 leader_uuid: "6106479de57e468193ec89ba032d4ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6106479de57e468193ec89ba032d4ad2" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 44625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 } health_report { overall_health: UNKNOWN } } }
I20260619 15:01:01.627511 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.624167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.10.71.131:44625)
0619 15:01:01.624267 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.624269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.624280 (+ 11us) server_negotiation.cc:407] Connection header received
0619 15:01:01.624433 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.624436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.624474 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.624546 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:01.624750 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.625209 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.626372 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.626550 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.626618 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.626836 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:01.627154 (+ 318us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:01.627242 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.627313 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.627358 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 15:01:01.632258 28939 raft_consensus.cc:1261] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 1 FOLLOWER]: Refusing update from remote peer 6106479de57e468193ec89ba032d4ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:01.632258 28805 raft_consensus.cc:1261] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 1 FOLLOWER]: Refusing update from remote peer 6106479de57e468193ec89ba032d4ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:01.632848 29146 consensus_queue.cc:1035] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7200069c89b54b738dd32456dc1c8743" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:01.632977 29144 consensus_queue.cc:1035] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "669d20c78f004efbb907c593c97e6964" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:01.639302 29149 mvcc.cc:205] Tried to move back new op lower bound from 7298585647639703552 to 7298585647530692608. Current Snapshot: MvccSnapshot[applied={T|T < 7298585647639703552}]
I20260619 15:01:01.639659 29150 mvcc.cc:205] Tried to move back new op lower bound from 7298585647639703552 to 7298585647530692608. Current Snapshot: MvccSnapshot[applied={T|T < 7298585647639703552}]
I20260619 15:01:01.640935 29151 mvcc.cc:205] Tried to move back new op lower bound from 7298585647639703552 to 7298585647530692608. Current Snapshot: MvccSnapshot[applied={T|T < 7298585647639703552 or (T in {7298585647639703552})}]
I20260619 15:01:01.647249 28658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:01.647394 28658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:01.649590 28658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24db191cf634f27a1b2829a0166f69d: Sending DeleteTablet for 3 replicas of tablet 0dd44cbae30144d98b9c0ef8f2afcea3
I20260619 15:01:01.650259 29053 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd44cbae30144d98b9c0ef8f2afcea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:01 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:35898
I20260619 15:01:01.650293 28919 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd44cbae30144d98b9c0ef8f2afcea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:01 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:37404
I20260619 15:01:01.650571 28785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd44cbae30144d98b9c0ef8f2afcea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:01 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:36502
I20260619 15:01:01.650739 29160 tablet_replica.cc:330] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: stopping tablet replica
I20260619 15:01:01.650779 29159 tablet_replica.cc:330] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: stopping tablet replica
I20260619 15:01:01.651022 29159 raft_consensus.cc:2229] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:01:01.651017 29160 raft_consensus.cc:2229] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:01:01.651242 29161 tablet_replica.cc:330] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964: stopping tablet replica
I20260619 15:01:01.651309 29160 raft_consensus.cc:2258] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:01.651450 29159 raft_consensus.cc:2258] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:01.651567 29161 raft_consensus.cc:2229] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 669d20c78f004efbb907c593c97e6964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:01:01.651598 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28722
I20260619 15:01:01.653566 29159 ts_tablet_manager.cc:1905] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:01.653744 29160 ts_tablet_manager.cc:1905] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:01.656093 29159 ts_tablet_manager.cc:1918] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:01.656229 29159 log.cc:1196] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/0dd44cbae30144d98b9c0ef8f2afcea3
I20260619 15:01:01.656642 29159 ts_tablet_manager.cc:1939] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 6106479de57e468193ec89ba032d4ad2: Deleting consensus metadata
I20260619 15:01:01.656816 29160 ts_tablet_manager.cc:1918] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:01.656936 29160 log.cc:1196] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/0dd44cbae30144d98b9c0ef8f2afcea3
I20260619 15:01:01.657258 29160 ts_tablet_manager.cc:1939] T 0dd44cbae30144d98b9c0ef8f2afcea3 P 7200069c89b54b738dd32456dc1c8743: Deleting consensus metadata
I20260619 15:01:01.657758 28647 catalog_manager.cc:4841] TS 6106479de57e468193ec89ba032d4ad2 (127.10.71.131:44625): tablet 0dd44cbae30144d98b9c0ef8f2afcea3 (table test-table [id=b380dbb8e5be48fcbfd3d7aa00a3bb99]) successfully deleted
I20260619 15:01:01.658381 28646 catalog_manager.cc:4841] TS 7200069c89b54b738dd32456dc1c8743 (127.10.71.130:37095): tablet 0dd44cbae30144d98b9c0ef8f2afcea3 (table test-table [id=b380dbb8e5be48fcbfd3d7aa00a3bb99]) successfully deleted
W20260619 15:01:01.659188 28646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:37175 (error 108)
I20260619 15:01:01.659919 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28856
I20260619 15:01:01.660099 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.659830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:37175 (local address 127.0.0.1:36510)
0619 15:01:01.659935 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:01.660016 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 15:01:01.660523 28646 catalog_manager.cc:4584] TS 669d20c78f004efbb907c593c97e6964 (127.10.71.129:37175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dd44cbae30144d98b9c0ef8f2afcea3: Network error: Client connection negotiation failed: client connection to 127.10.71.129:37175: connect: Connection refused (error 111)
I20260619 15:01:01.665207 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28990
I20260619 15:01:01.669451 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 28628
2026-06-19T15:01:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T15:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:01Z Disabled control of system clock
I20260619 15:01:01.697161 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38161
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:01:01.793931 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:01.794137 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:01.794166 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:01.797632 29166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:01.797669 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:01.797681 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:01.797695 29166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:01.797708 29166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:01.801807 29166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:38161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:01.802642 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:01.803710 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:01.811096 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:01.811106 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.811293 29166 server_base.cc:1034] running on GCE node
W20260619 15:01:01.811105 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:01.811928 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:01.812934 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:01.814111 29166 hybrid_clock.cc:648] HybridClock initialized: now 1781881261814080 us; error 43 us; skew 500 ppm
I20260619 15:01:01.815752 29166 webserver.cc:460] Webserver started at http://127.10.71.190:40603/ using document root <none> and password file <none>
I20260619 15:01:01.816246 29166 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:01.816349 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:01.816572 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:01.818249 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "4bd3562d17df41ea92f6908ea8b4ec9a"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "c5071504b979f8a4f0ae2f404a4becc6"
server_key_iv: "0437e69ec1d4645e28b020e42aa33a4d"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.818766 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "4bd3562d17df41ea92f6908ea8b4ec9a"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "c5071504b979f8a4f0ae2f404a4becc6"
server_key_iv: "0437e69ec1d4645e28b020e42aa33a4d"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.822038 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:01.824030 29180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:01.825084 29166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:01.825208 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "4bd3562d17df41ea92f6908ea8b4ec9a"
format_stamp: "Formatted at 2026-06-19 15:01:01 on dist-test-slave-3qdh"
server_key: "c5071504b979f8a4f0ae2f404a4becc6"
server_key_iv: "0437e69ec1d4645e28b020e42aa33a4d"
server_key_version: "encryptionkey@0"
I20260619 15:01:01.825321 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:01.833926 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:01.834434 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:01.834607 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:01.840057 29166 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:38161
I20260619 15:01:01.840087 29232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:38161 every 8 connection(s)
I20260619 15:01:01.840848 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:01.841861 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29166
I20260619 15:01:01.841980 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:01.842268 10526 external_mini_cluster.cc:1428] Setting key ef2d3f2e9353d28eda84056a6061c6ec
I20260619 15:01:01.844055 29233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:01.848917 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a: Bootstrap starting.
I20260619 15:01:01.850483 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:01.843422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.10.71.190:38161)
0619 15:01:01.843975 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:01.843988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:01.844016 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:01:01.844666 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:01.844681 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:01.844999 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:01.845292 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:01.845646 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.846486 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.847322 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:01.847753 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:01.847917 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:01.848816 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:01.848821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:01.848831 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:01.849287 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:01.849341 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:01.849362 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:01.849453 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:01.849617 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 15:01:01.849903 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260619 15:01:01.850934 29233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:01.851620 29233 log.cc:824] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:01.853307 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a: No bootstrap required, opened a new log
I20260619 15:01:01.856173 29233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38161 } }
I20260619 15:01:01.856420 29233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:01.856488 29233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd3562d17df41ea92f6908ea8b4ec9a, State: Initialized, Role: FOLLOWER
I20260619 15:01:01.856856 29233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38161 } }
I20260619 15:01:01.856989 29233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:01.857050 29233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:01.857133 29233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:01.857867 29233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38161 } }
I20260619 15:01:01.858119 29233 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd3562d17df41ea92f6908ea8b4ec9a; no voters:
I20260619 15:01:01.858315 29233 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:01.858426 29238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:01.858632 29238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [term 1 LEADER]: Becoming Leader. State: Replica: 4bd3562d17df41ea92f6908ea8b4ec9a, State: Running, Role: LEADER
I20260619 15:01:01.858896 29238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38161 } }
I20260619 15:01:01.859314 29233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:01.860291 29240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd3562d17df41ea92f6908ea8b4ec9a. Latest consensus state: current_term: 1 leader_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38161 } } }
I20260619 15:01:01.860426 29240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:01.860673 29239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd3562d17df41ea92f6908ea8b4ec9a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38161 } } }
I20260619 15:01:01.860755 29239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:01.860870 29247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:01.863687 29247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:01.868403 29247 catalog_manager.cc:1347] Generated new cluster ID: 923963846e7544189fc673cd22d92bd3
I20260619 15:01:01.868463 29247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:01.873718 29247 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:01.874424 29247 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:01.877871 29247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a: Generated new TSK 0
I20260619 15:01:01.878397 29247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:01.885313 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38161
--builtin_ntp_servers=127.10.71.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:01:01.985392 29257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:01.985636 29257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:01.985671 29257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:01.989498 29257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:01.989548 29257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:01.989632 29257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:01:01.994021 29257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:01.994954 29257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:01.996290 29257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:02.005345 29263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:02.005422 29257 server_base.cc:1034] running on GCE node
W20260619 15:01:02.005414 29265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:02.005432 29262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:02.005899 29257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:02.006457 29257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:02.007635 29257 hybrid_clock.cc:648] HybridClock initialized: now 1781881262007621 us; error 28 us; skew 500 ppm
I20260619 15:01:02.009265 29257 webserver.cc:460] Webserver started at http://127.10.71.129:41719/ using document root <none> and password file <none>
I20260619 15:01:02.009742 29257 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:02.009819 29257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:02.010016 29257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:02.011808 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "0fdadbc3b42d4e01a28e0152a6569ab8"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "3187bead167e7dc69930379d4265522e"
server_key_iv: "2f7e886d7e7baddae6359ff9af249c3f"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.012414 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "0fdadbc3b42d4e01a28e0152a6569ab8"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "3187bead167e7dc69930379d4265522e"
server_key_iv: "2f7e886d7e7baddae6359ff9af249c3f"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.015626 29257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:01:02.017737 29271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.018778 29257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:02.018901 29257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "0fdadbc3b42d4e01a28e0152a6569ab8"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "3187bead167e7dc69930379d4265522e"
server_key_iv: "2f7e886d7e7baddae6359ff9af249c3f"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.019007 29257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:02.042686 29257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:02.043170 29257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:02.043329 29257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:02.043778 29257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:02.044559 29257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:02.044618 29257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.044680 29257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:02.044715 29257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.052407 29257 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:42551
I20260619 15:01:02.052441 29384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:42551 every 8 connection(s)
I20260619 15:01:02.053155 29257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 15:01:02.060770 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29257
I20260619 15:01:02.060802 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.054659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:49717 (local address 127.10.71.190:38161)
0619 15:01:02.054793 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.054797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.055636 (+ 839us) server_negotiation.cc:407] Connection header received
0619 15:01:02.056861 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.056864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.056904 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.056969 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:02.057682 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.058155 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.059108 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.059306 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.059371 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.060256 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:02.060259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:02.060261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:02.060270 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:02.060286 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:02.060326 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:02.060417 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.060581 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.060669 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 15:01:02.060890 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 15:01:02.061192 10526 external_mini_cluster.cc:1428] Setting key 1bad94873c5457ecb31a1db7684f7804
I20260619 15:01:02.061595 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.054953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38161 (local address 127.10.71.129:49717)
0619 15:01:02.055505 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.055539 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.056701 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.057111 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.057117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.057188 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:02.057575 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.057585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.058271 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.058273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.059020 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.059027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.059406 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.059408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.059623 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.060051 (+ 428us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:01:02.060052 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:02.060083 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:01:02.060134 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:02.060421 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:02.060427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:02.060431 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.060548 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.060769 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
I20260619 15:01:02.062666 29385 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38161
I20260619 15:01:02.062853 29385 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:02.063063 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38161
--builtin_ntp_servers=127.10.71.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:01:02.063313 29385 heartbeater.cc:511] Master 127.10.71.190:38161 requested a full tablet report, sending...
I20260619 15:01:02.064673 29197 ts_manager.cc:194] Registered new tserver with Master: 0fdadbc3b42d4e01a28e0152a6569ab8 (127.10.71.129:42551)
I20260619 15:01:02.065639 29197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.129:49717
W20260619 15:01:02.171823 29388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:02.172058 29388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:02.172138 29388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:02.175673 29388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:02.175716 29388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:02.175762 29388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:01:02.179952 29388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:02.180814 29388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:02.182030 29388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:02.191103 29393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:02.191118 29396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:02.191123 29394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:02.191525 29388 server_base.cc:1034] running on GCE node
I20260619 15:01:02.191927 29388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:02.192560 29388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:02.193701 29388 hybrid_clock.cc:648] HybridClock initialized: now 1781881262193680 us; error 36 us; skew 500 ppm
I20260619 15:01:02.195232 29388 webserver.cc:460] Webserver started at http://127.10.71.130:35163/ using document root <none> and password file <none>
I20260619 15:01:02.195722 29388 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:02.195778 29388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:02.195998 29388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:02.197754 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "ef940d5ae0fe4044b46ce0381428c746"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "088d284880a699a607668aa040509c27"
server_key_iv: "7e967e89d4f8e7bd1e77ad48c2adb1ed"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.198241 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "ef940d5ae0fe4044b46ce0381428c746"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "088d284880a699a607668aa040509c27"
server_key_iv: "7e967e89d4f8e7bd1e77ad48c2adb1ed"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.201313 29388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:02.203351 29402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.204382 29388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:01:02.204483 29388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "ef940d5ae0fe4044b46ce0381428c746"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "088d284880a699a607668aa040509c27"
server_key_iv: "7e967e89d4f8e7bd1e77ad48c2adb1ed"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.204566 29388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:02.217571 29388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:02.218061 29388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:02.218219 29388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:02.218684 29388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:02.219386 29388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:02.219425 29388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.219462 29388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:02.219476 29388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.228341 29388 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:43871
I20260619 15:01:02.228396 29515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:43871 every 8 connection(s)
I20260619 15:01:02.229094 29388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:01:02.236562 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.230538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:40583 (local address 127.10.71.190:38161)
0619 15:01:02.230682 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.230685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.231551 (+ 866us) server_negotiation.cc:407] Connection header received
0619 15:01:02.232813 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.232815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.232858 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.232921 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:02.233632 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.234076 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.235027 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.235215 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.235288 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.236150 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:02.236152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:02.236154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:02.236164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:02.236180 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:02.236186 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:02.236237 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.236405 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.236464 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 15:01:02.237237 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.230805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38161 (local address 127.10.71.130:40583)
0619 15:01:02.231404 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.231452 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.232653 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.233059 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.233064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.233130 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:02.233519 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.233530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.234169 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.234171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.234911 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.234919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.235332 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.235334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.235463 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.235934 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:01:02.235936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:02.235970 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:01:02.236020 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:02.236253 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:02.236259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:02.236262 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.236376 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.236555 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260619 15:01:02.238300 29516 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38161
I20260619 15:01:02.238509 29516 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:02.238684 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29388
I20260619 15:01:02.238794 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 15:01:02.238978 29516 heartbeater.cc:511] Master 127.10.71.190:38161 requested a full tablet report, sending...
I20260619 15:01:02.239034 10526 external_mini_cluster.cc:1428] Setting key 22a70262aa8cb38c2d4ca08a6a7ab60d
I20260619 15:01:02.239969 29196 ts_manager.cc:194] Registered new tserver with Master: ef940d5ae0fe4044b46ce0381428c746 (127.10.71.130:43871)
I20260619 15:01:02.240617 29196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.130:40583
I20260619 15:01:02.240965 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38161
--builtin_ntp_servers=127.10.71.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:01:02.338011 29519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:02.338209 29519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:02.338241 29519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:02.342159 29519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:02.342206 29519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:02.342257 29519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:01:02.346731 29519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:38161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:02.347553 29519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:02.348809 29519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:02.357753 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:02.357750 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:02.357746 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:02.357923 29519 server_base.cc:1034] running on GCE node
I20260619 15:01:02.358343 29519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:02.358862 29519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:02.360035 29519 hybrid_clock.cc:648] HybridClock initialized: now 1781881262360018 us; error 33 us; skew 500 ppm
I20260619 15:01:02.362089 29519 webserver.cc:460] Webserver started at http://127.10.71.131:40143/ using document root <none> and password file <none>
I20260619 15:01:02.362635 29519 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:02.362699 29519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:02.362937 29519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:02.364840 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "f96c2c0812e04c48aa0232fa319ed66c"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "aa03ba2b823f939d167d77185c2737d9"
server_key_iv: "f2a0fcd17c5873e397e7bec0dd821dc7"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.365370 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "f96c2c0812e04c48aa0232fa319ed66c"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "aa03ba2b823f939d167d77185c2737d9"
server_key_iv: "f2a0fcd17c5873e397e7bec0dd821dc7"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.368749 29519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:02.370977 29533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.372201 29519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:02.372426 29519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "f96c2c0812e04c48aa0232fa319ed66c"
format_stamp: "Formatted at 2026-06-19 15:01:02 on dist-test-slave-3qdh"
server_key: "aa03ba2b823f939d167d77185c2737d9"
server_key_iv: "f2a0fcd17c5873e397e7bec0dd821dc7"
server_key_version: "encryptionkey@0"
I20260619 15:01:02.372576 29519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:02.386770 29519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:02.387303 29519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:02.387475 29519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:02.387951 29519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:02.388736 29519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:02.388803 29519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.388861 29519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:02.388906 29519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:02.396678 29519 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:37217
I20260619 15:01:02.396770 29646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:37217 every 8 connection(s)
I20260619 15:01:02.397462 29519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 15:01:02.405220 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.398975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:57479 (local address 127.10.71.190:38161)
0619 15:01:02.399127 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.399132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.399975 (+ 843us) server_negotiation.cc:407] Connection header received
0619 15:01:02.401216 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.401219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.401263 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.401335 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:02.402220 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.402696 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.403674 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.403865 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.403935 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.404857 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:02.404859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:02.404861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:02.404871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:02.404887 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:02.404894 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:02.404946 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.405076 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.405120 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 15:01:02.405963 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.399302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:38161 (local address 127.10.71.131:57479)
0619 15:01:02.399852 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.399884 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.401029 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.401482 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.401490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.401566 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:02.402104 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.402118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.402808 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.402810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.403579 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.403587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.403978 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.403980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.404106 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.404654 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:01:02.404655 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:02.404688 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:01:02.404740 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:02.404961 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:02.404967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:02.404970 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.405061 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.405236 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
I20260619 15:01:02.406164 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29519
I20260619 15:01:02.406265 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 15:01:02.406554 10526 external_mini_cluster.cc:1428] Setting key 80299001a815b9b73c575d32760d1df3
I20260619 15:01:02.407011 29647 heartbeater.cc:347] Connected to a master server at 127.10.71.190:38161
I20260619 15:01:02.407212 29647 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:02.407754 29647 heartbeater.cc:511] Master 127.10.71.190:38161 requested a full tablet report, sending...
I20260619 15:01:02.408668 29196 ts_manager.cc:194] Registered new tserver with Master: f96c2c0812e04c48aa0232fa319ed66c (127.10.71.131:37217)
I20260619 15:01:02.409139 29196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.131:57479
I20260619 15:01:02.419862 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:01:02.424886 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.421586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.10.71.190:38161)
0619 15:01:02.421743 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.421747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.421876 (+ 129us) server_negotiation.cc:407] Connection header received
0619 15:01:02.421958 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.421960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.421992 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.422058 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:02.422287 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.422712 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.423444 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.423624 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.423702 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.424528 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:02.424530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:02.424532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:02.424543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:02.424560 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:02.424567 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:02.424659 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.424773 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.424798 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 15:01:02.427296 29196 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:01:02.430147 29196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:01:02.443032 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.437648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43871 (local address 127.0.0.1:45748)
0619 15:01:02.438282 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.438294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.438369 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.439400 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.439403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.439421 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:02.439491 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.439497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.440723 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.440726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.441773 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.441805 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.442555 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.442558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.442649 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.442699 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.442812 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.442910 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":78,"threads_started":1}
I20260619 15:01:02.443063 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.437946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.10.71.130:43871)
0619 15:01:02.438936 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.438949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.438969 (+ 20us) server_negotiation.cc:407] Connection header received
0619 15:01:02.439036 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.439042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.439205 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.439404 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:02.439596 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.440620 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.441921 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.442438 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.442567 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.442768 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.442844 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.442928 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
I20260619 15:01:02.443456 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.437384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:42551 (local address 127.0.0.1:46206)
0619 15:01:02.438056 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.438104 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.438222 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.438909 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.438914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.438935 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:02.439070 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.439076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.440481 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.440484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.441711 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.441722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.442969 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.442973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.443135 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.443186 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.443259 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.443316 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":109,"thread_start_us":79,"threads_started":1}
I20260619 15:01:02.443754 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.438335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.10.71.131:37217)
0619 15:01:02.438710 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.438721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.438745 (+ 24us) server_negotiation.cc:407] Connection header received
0619 15:01:02.438792 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.438795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.438955 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.439094 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:02.439300 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.440103 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.441947 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.442639 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.442764 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.442909 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.443537 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.443622 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260619 15:01:02.443918 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.437528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.10.71.129:42551)
0619 15:01:02.437998 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.438010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.438352 (+ 342us) server_negotiation.cc:407] Connection header received
0619 15:01:02.438430 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.438434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.438617 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.438754 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:02.439190 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.440376 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.441838 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.442633 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.442764 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.442909 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.443681 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.443767 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":126,"threads_started":1}
I20260619 15:01:02.444118 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.437992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37217 (local address 127.0.0.1:49842)
0619 15:01:02.438549 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.438560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.438636 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.439077 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.439080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.439096 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:02.439180 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.439185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.440211 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.440214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.441718 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.441723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.442969 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.442973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.443409 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.443427 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.443497 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.443708 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":95,"thread_start_us":90,"threads_started":1}
I20260619 15:01:02.445855 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0033efcd184d91bc4a6a8322b1761a (DEFAULT_TABLE table=test-table [id=a1532433364f41f58351e9cfe01a221a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:02.446221 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0033efcd184d91bc4a6a8322b1761a (DEFAULT_TABLE table=test-table [id=a1532433364f41f58351e9cfe01a221a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:02.446310 29319 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0033efcd184d91bc4a6a8322b1761a (DEFAULT_TABLE table=test-table [id=a1532433364f41f58351e9cfe01a221a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:02.446852 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0033efcd184d91bc4a6a8322b1761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:02.447140 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0033efcd184d91bc4a6a8322b1761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:02.447191 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0033efcd184d91bc4a6a8322b1761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:02.452437 29664 tablet_bootstrap.cc:492] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Bootstrap starting.
I20260619 15:01:02.453913 29665 tablet_bootstrap.cc:492] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Bootstrap starting.
I20260619 15:01:02.455263 29664 tablet_bootstrap.cc:654] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:02.455415 29665 tablet_bootstrap.cc:654] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:02.455950 29664 log.cc:824] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:02.456007 29665 log.cc:824] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:02.456578 29666 tablet_bootstrap.cc:492] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: Bootstrap starting.
I20260619 15:01:02.457566 29664 tablet_bootstrap.cc:492] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: No bootstrap required, opened a new log
I20260619 15:01:02.457758 29664 ts_tablet_manager.cc:1397] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 15:01:02.458137 29666 tablet_bootstrap.cc:654] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:02.458906 29666 log.cc:824] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:02.460614 29666 tablet_bootstrap.cc:492] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: No bootstrap required, opened a new log
I20260619 15:01:02.460788 29666 ts_tablet_manager.cc:1397] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 15:01:02.461287 29665 tablet_bootstrap.cc:492] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: No bootstrap required, opened a new log
I20260619 15:01:02.461474 29665 ts_tablet_manager.cc:1397] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260619 15:01:02.463814 29666 raft_consensus.cc:348] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.464013 29666 raft_consensus.cc:374] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:02.464073 29666 raft_consensus.cc:729] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdadbc3b42d4e01a28e0152a6569ab8, State: Initialized, Role: FOLLOWER
I20260619 15:01:02.464391 29665 raft_consensus.cc:348] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.464509 29666 consensus_queue.cc:260] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.464624 29665 raft_consensus.cc:374] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:02.464676 29665 raft_consensus.cc:729] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef940d5ae0fe4044b46ce0381428c746, State: Initialized, Role: FOLLOWER
I20260619 15:01:02.464896 29664 raft_consensus.cc:348] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.465098 29385 heartbeater.cc:503] Master 127.10.71.190:38161 was elected leader, sending a full tablet report...
I20260619 15:01:02.465117 29664 raft_consensus.cc:374] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:02.465160 29664 raft_consensus.cc:729] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96c2c0812e04c48aa0232fa319ed66c, State: Initialized, Role: FOLLOWER
I20260619 15:01:02.465466 29666 ts_tablet_manager.cc:1428] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:01:02.465512 29664 consensus_queue.cc:260] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.466253 29647 heartbeater.cc:503] Master 127.10.71.190:38161 was elected leader, sending a full tablet report...
I20260619 15:01:02.466228 29665 consensus_queue.cc:260] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.466393 29664 ts_tablet_manager.cc:1428] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260619 15:01:02.466905 29516 heartbeater.cc:503] Master 127.10.71.190:38161 was elected leader, sending a full tablet report...
I20260619 15:01:02.467077 29665 ts_tablet_manager.cc:1428] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260619 15:01:02.479810 29517 tablet.cc:2369] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:01:02.554194 29386 tablet.cc:2369] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 15:01:02.648386 29648 tablet.cc:2369] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:02.765399 29670 raft_consensus.cc:482] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:01:02.765626 29670 raft_consensus.cc:504] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.766716 29670 leader_election.cc:288] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef940d5ae0fe4044b46ce0381428c746 (127.10.71.130:43871), f96c2c0812e04c48aa0232fa319ed66c (127.10.71.131:37217)
I20260619 15:01:02.770591 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.767034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:44897 (local address 127.10.71.130:43871)
0619 15:01:02.767156 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.767160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.767373 (+ 213us) server_negotiation.cc:407] Connection header received
0619 15:01:02.767512 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.767514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.767553 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.767620 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:02.767892 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.768458 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.769639 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.769988 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.770105 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.770166 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.770407 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.770475 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 15:01:02.770591 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.767078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:46109 (local address 127.10.71.131:37217)
0619 15:01:02.767191 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.767195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.767325 (+ 130us) server_negotiation.cc:407] Connection header received
0619 15:01:02.767477 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.767480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.767517 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.767585 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:02.767875 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.768465 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.769641 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.769998 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.770110 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.770171 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.770398 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.770467 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 15:01:02.770625 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.766945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:37217 (local address 127.10.71.129:46109)
0619 15:01:02.767244 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.767255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.767375 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.767608 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.767610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.767624 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:02.767770 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.767775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.768572 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.768577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.769527 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.769535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.770107 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.770110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.770272 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.770301 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.770382 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.770476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":91,"thread_start_us":67,"threads_started":1}
I20260619 15:01:02.770629 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.766944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:43871 (local address 127.10.71.129:44897)
0619 15:01:02.767288 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.767298 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:02.767409 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:02.767641 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:02.767643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:02.767657 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:02.767793 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.767798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.768567 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.768570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.769524 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:02.769532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.770099 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.770102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:02.770281 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.770314 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 15:01:02.770394 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 15:01:02.770480 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":178,"thread_start_us":84,"threads_started":1}
I20260619 15:01:02.771190 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0033efcd184d91bc4a6a8322b1761a" candidate_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96c2c0812e04c48aa0232fa319ed66c" is_pre_election: true
I20260619 15:01:02.771476 29601 raft_consensus.cc:2454] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fdadbc3b42d4e01a28e0152a6569ab8 in term 0.
I20260619 15:01:02.771587 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0033efcd184d91bc4a6a8322b1761a" candidate_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef940d5ae0fe4044b46ce0381428c746" is_pre_election: true
I20260619 15:01:02.771875 29470 raft_consensus.cc:2454] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fdadbc3b42d4e01a28e0152a6569ab8 in term 0.
I20260619 15:01:02.771940 29272 leader_election.cc:302] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdadbc3b42d4e01a28e0152a6569ab8, f96c2c0812e04c48aa0232fa319ed66c; no voters:
I20260619 15:01:02.772228 29670 raft_consensus.cc:2781] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:01:02.772338 29670 raft_consensus.cc:482] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:01:02.772401 29670 raft_consensus.cc:3037] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:02.773315 29670 raft_consensus.cc:504] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.773623 29670 leader_election.cc:288] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [CANDIDATE]: Term 1 election: Requested vote from peers ef940d5ae0fe4044b46ce0381428c746 (127.10.71.130:43871), f96c2c0812e04c48aa0232fa319ed66c (127.10.71.131:37217)
I20260619 15:01:02.773914 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0033efcd184d91bc4a6a8322b1761a" candidate_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96c2c0812e04c48aa0232fa319ed66c"
I20260619 15:01:02.773885 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0033efcd184d91bc4a6a8322b1761a" candidate_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef940d5ae0fe4044b46ce0381428c746"
I20260619 15:01:02.774016 29601 raft_consensus.cc:3037] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:02.774010 29470 raft_consensus.cc:3037] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:02.775066 29470 raft_consensus.cc:2454] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fdadbc3b42d4e01a28e0152a6569ab8 in term 1.
I20260619 15:01:02.775066 29601 raft_consensus.cc:2454] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fdadbc3b42d4e01a28e0152a6569ab8 in term 1.
I20260619 15:01:02.775346 29274 leader_election.cc:302] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fdadbc3b42d4e01a28e0152a6569ab8, ef940d5ae0fe4044b46ce0381428c746; no voters:
I20260619 15:01:02.775494 29670 raft_consensus.cc:2781] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:02.775681 29670 raft_consensus.cc:686] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [term 1 LEADER]: Becoming Leader. State: Replica: 0fdadbc3b42d4e01a28e0152a6569ab8, State: Running, Role: LEADER
I20260619 15:01:02.775987 29670 consensus_queue.cc:237] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } }
I20260619 15:01:02.778782 29195 catalog_manager.cc:5495] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fdadbc3b42d4e01a28e0152a6569ab8 (127.10.71.129). New cstate: current_term: 1 leader_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fdadbc3b42d4e01a28e0152a6569ab8" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 } health_report { overall_health: UNKNOWN } } }
I20260619 15:01:02.800814 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.797556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.10.71.129:42551)
0619 15:01:02.797662 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:02.797668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:02.797683 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:01:02.797871 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:02.797874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:02.797920 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:02.798036 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:02.798257 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.798703 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.799477 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:02.799671 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:02.799765 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:02.800074 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:02.800425 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:02.800578 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:02.800660 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 15:01:02.800690 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 15:01:02.805764 29601 raft_consensus.cc:1261] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 1 FOLLOWER]: Refusing update from remote peer 0fdadbc3b42d4e01a28e0152a6569ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:02.805787 29470 raft_consensus.cc:1261] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 1 FOLLOWER]: Refusing update from remote peer 0fdadbc3b42d4e01a28e0152a6569ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:02.806321 29670 consensus_queue.cc:1035] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96c2c0812e04c48aa0232fa319ed66c" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:02.806493 29675 consensus_queue.cc:1035] T 3d0033efcd184d91bc4a6a8322b1761a P 0fdadbc3b42d4e01a28e0152a6569ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef940d5ae0fe4044b46ce0381428c746" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 43871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:02.818222 29679 mvcc.cc:205] Tried to move back new op lower bound from 7298585652446171136 to 7298585652333039616. Current Snapshot: MvccSnapshot[applied={T|T < 7298585652446171136}]
I20260619 15:01:02.823943 29197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:02.824118 29197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:02.826300 29197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bd3562d17df41ea92f6908ea8b4ec9a: Sending DeleteTablet for 3 replicas of tablet 3d0033efcd184d91bc4a6a8322b1761a
I20260619 15:01:02.827041 29581 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d0033efcd184d91bc4a6a8322b1761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:02 UTC) from {username='slave'} at 127.0.0.1:49842
I20260619 15:01:02.827451 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29257
I20260619 15:01:02.827490 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d0033efcd184d91bc4a6a8322b1761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:02 UTC) from {username='slave'} at 127.0.0.1:45748
I20260619 15:01:02.828802 29688 tablet_replica.cc:330] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: stopping tablet replica
I20260619 15:01:02.829088 29688 raft_consensus.cc:2229] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:01:02.829326 29688 raft_consensus.cc:2258] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:02.830323 29688 ts_tablet_manager.cc:1905] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:02.832388 29688 ts_tablet_manager.cc:1918] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:02.832480 29688 log.cc:1196] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/3d0033efcd184d91bc4a6a8322b1761a
I20260619 15:01:02.832809 29688 ts_tablet_manager.cc:1939] T 3d0033efcd184d91bc4a6a8322b1761a P ef940d5ae0fe4044b46ce0381428c746: Deleting consensus metadata
I20260619 15:01:02.832959 29687 tablet_replica.cc:330] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: stopping tablet replica
I20260619 15:01:02.833209 29687 raft_consensus.cc:2229] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 15:01:02.833350 29181 connection.cc:535] client connection to 127.10.71.129:42551 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:01:02.833478 29687 raft_consensus.cc:2258] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 15:01:02.833470 29181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 15:01:02.833833 29183 catalog_manager.cc:4841] TS ef940d5ae0fe4044b46ce0381428c746 (127.10.71.130:43871): tablet 3d0033efcd184d91bc4a6a8322b1761a (table test-table [id=a1532433364f41f58351e9cfe01a221a]) successfully deleted
I20260619 15:01:02.834426 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:02.834170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:42551 (local address 127.0.0.1:46236)
0619 15:01:02.834290 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:02.834355 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 15:01:02.834431 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29388
I20260619 15:01:02.835356 29687 ts_tablet_manager.cc:1905] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 15:01:02.836526 29181 catalog_manager.cc:4584] TS 0fdadbc3b42d4e01a28e0152a6569ab8 (127.10.71.129:42551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d0033efcd184d91bc4a6a8322b1761a: Network error: Client connection negotiation failed: client connection to 127.10.71.129:42551: connect: Connection refused (error 111)
I20260619 15:01:02.837472 29687 ts_tablet_manager.cc:1918] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:02.837575 29687 log.cc:1196] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/3d0033efcd184d91bc4a6a8322b1761a
I20260619 15:01:02.837816 29687 ts_tablet_manager.cc:1939] T 3d0033efcd184d91bc4a6a8322b1761a P f96c2c0812e04c48aa0232fa319ed66c: Deleting consensus metadata
I20260619 15:01:02.838733 29181 catalog_manager.cc:4841] TS f96c2c0812e04c48aa0232fa319ed66c (127.10.71.131:37217): tablet 3d0033efcd184d91bc4a6a8322b1761a (table test-table [id=a1532433364f41f58351e9cfe01a221a]) successfully deleted
I20260619 15:01:02.839732 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29519
I20260619 15:01:02.846129 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29166
2026-06-19T15:01:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:02 dist-test-slave-3qdh krb5kdc[29692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:02 dist-test-slave-3qdh krb5kdc[29692](info): set up 2 sockets
Jun 19 15:01:02 dist-test-slave-3qdh krb5kdc[29692](info): commencing operation
krb5kdc: starting...
W20260619 15:01:03.800340 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:03 dist-test-slave-3qdh krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:03.901136 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42955
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:42955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:03.996071 29705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:03.996284 29705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:03.996357 29705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:03.999922 29705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:04.000013 29705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:04.000061 29705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:04.000098 29705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:04.000133 29705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:04.005132 29705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:42955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:42955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:04.006013 29705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:04.007084 29705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:04.014444 29710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:04.014454 29711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:04.014577 29705 server_base.cc:1034] running on GCE node
W20260619 15:01:04.014454 29713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:04.015133 29705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:04.016031 29705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:04.017195 29705 hybrid_clock.cc:648] HybridClock initialized: now 1781881264017180 us; error 25 us; skew 500 ppm
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:04.019503 29705 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:04.020187 29705 webserver.cc:460] Webserver started at http://127.10.71.190:34193/ using document root <none> and password file <none>
I20260619 15:01:04.020678 29705 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:04.020759 29705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:04.020946 29705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:04.022691 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "da61939fc66149a28aee4dd5f8f0c095"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "4806a20ccf399ce2ffafdb302e03f7e2"
server_key_iv: "758c6c012680b47581b77be89ae3caf7"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.023159 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "da61939fc66149a28aee4dd5f8f0c095"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "4806a20ccf399ce2ffafdb302e03f7e2"
server_key_iv: "758c6c012680b47581b77be89ae3caf7"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.026099 29705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:04.028045 29720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.029093 29705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:04.029217 29705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "da61939fc66149a28aee4dd5f8f0c095"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "4806a20ccf399ce2ffafdb302e03f7e2"
server_key_iv: "758c6c012680b47581b77be89ae3caf7"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.029311 29705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:04.053119 29705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:04.055670 29705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:04.055838 29705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:04.061779 29705 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:42955
I20260619 15:01:04.061829 29772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:42955 every 8 connection(s)
I20260619 15:01:04.062608 29705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:04.064808 29773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:04.065665 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29705
I20260619 15:01:04.065793 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:04.066083 10526 external_mini_cluster.cc:1428] Setting key 622c8826e513b6c8d585f11a0429ddc8
I20260619 15:01:04.069926 29773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095: Bootstrap starting.
I20260619 15:01:04.071815 29773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:04.072571 29773 log.cc:824] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:04.074195 29773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095: No bootstrap required, opened a new log
I20260619 15:01:04.076905 29773 raft_consensus.cc:348] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61939fc66149a28aee4dd5f8f0c095" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42955 } }
I20260619 15:01:04.077093 29773 raft_consensus.cc:374] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:04.077144 29773 raft_consensus.cc:729] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da61939fc66149a28aee4dd5f8f0c095, State: Initialized, Role: FOLLOWER
I20260619 15:01:04.077564 29773 consensus_queue.cc:260] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61939fc66149a28aee4dd5f8f0c095" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42955 } }
I20260619 15:01:04.077695 29773 raft_consensus.cc:388] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:04.077773 29773 raft_consensus.cc:482] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:04.077867 29773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:04.078315 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.067224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.10.71.190:42955)
0619 15:01:04.067578 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.067590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.067617 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:01:04.068206 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.068221 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.068563 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.068847 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:04.069565 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.070434 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.071247 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.071707 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.071876 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.074269 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:04.074276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:04.074293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:04.074316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:04.075706 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.076103 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.076112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.076115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.076209 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.076489 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.076494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.076497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.076801 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:04.077005 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.077239 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.077469 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260619 15:01:04.078706 29773 raft_consensus.cc:504] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61939fc66149a28aee4dd5f8f0c095" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42955 } }
I20260619 15:01:04.079012 29773 leader_election.cc:302] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da61939fc66149a28aee4dd5f8f0c095; no voters:
I20260619 15:01:04.079295 29773 leader_election.cc:288] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:04.079382 29778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:04.079569 29778 raft_consensus.cc:686] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [term 1 LEADER]: Becoming Leader. State: Replica: da61939fc66149a28aee4dd5f8f0c095, State: Running, Role: LEADER
I20260619 15:01:04.079876 29778 consensus_queue.cc:237] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61939fc66149a28aee4dd5f8f0c095" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42955 } }
I20260619 15:01:04.080358 29773 sys_catalog.cc:564] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:04.081523 29780 sys_catalog.cc:455] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da61939fc66149a28aee4dd5f8f0c095. Latest consensus state: current_term: 1 leader_uuid: "da61939fc66149a28aee4dd5f8f0c095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61939fc66149a28aee4dd5f8f0c095" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42955 } } }
I20260619 15:01:04.081558 29779 sys_catalog.cc:455] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da61939fc66149a28aee4dd5f8f0c095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da61939fc66149a28aee4dd5f8f0c095" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 42955 } } }
I20260619 15:01:04.081727 29780 sys_catalog.cc:458] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:04.081755 29779 sys_catalog.cc:458] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:04.082504 29788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:04.084870 29788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:04.089700 29788 catalog_manager.cc:1347] Generated new cluster ID: e25e34b9af604e51b89c93b0234f8788
I20260619 15:01:04.089753 29788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:04.102020 29788 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:04.103049 29788 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:04.109642 29788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095: Generated new TSK 0
I20260619 15:01:04.110174 29788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
Entry for principal kudu/127.10.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab.
I20260619 15:01:04.141268 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42955
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:04.239576 29799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:04.239764 29799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:04.239799 29799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:04.243399 29799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:04.243443 29799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:04.243489 29799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:01:04.248270 29799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.129.keytab
--principal=kudu/127.10.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:04.249099 29799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:04.249977 29799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:04.258805 29807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:04.258811 29805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:04.258824 29799 server_base.cc:1034] running on GCE node
W20260619 15:01:04.258841 29804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:04.259363 29799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:04.259887 29799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:04.261061 29799 hybrid_clock.cc:648] HybridClock initialized: now 1781881264261044 us; error 27 us; skew 500 ppm
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:04.263486 29799 init.cc:377] Logged in from keytab as kudu/127.10.71.129@KRBTEST.COM (short username kudu)
I20260619 15:01:04.264204 29799 webserver.cc:460] Webserver started at http://127.10.71.129:34859/ using document root <none> and password file <none>
I20260619 15:01:04.264688 29799 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:04.264786 29799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:04.265009 29799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:04.266788 29799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "a6e90d02acbe4d528a55b77f103b7caf"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "2dc1efe91b0ee83d0c604380a49736b6"
server_key_iv: "4d35db06b0f63c54debffa5107e9d782"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.267274 29799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "a6e90d02acbe4d528a55b77f103b7caf"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "2dc1efe91b0ee83d0c604380a49736b6"
server_key_iv: "4d35db06b0f63c54debffa5107e9d782"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.270509 29799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:04.272670 29814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.273700 29799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:04.273826 29799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "a6e90d02acbe4d528a55b77f103b7caf"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "2dc1efe91b0ee83d0c604380a49736b6"
server_key_iv: "4d35db06b0f63c54debffa5107e9d782"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.273927 29799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:04.286242 29799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:04.288723 29799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:04.288923 29799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:04.289424 29799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:04.290126 29799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:04.290192 29799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.290256 29799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:04.290303 29799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.298125 29799 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:43361
I20260619 15:01:04.298151 29927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:43361 every 8 connection(s)
I20260619 15:01:04.298812 29799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 15:01:04.305925 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29799
I20260619 15:01:04.306072 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 15:01:04.306424 10526 external_mini_cluster.cc:1428] Setting key 07ebc5c33124c217264a69aa8ebd1c9c
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.129@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:04.311532 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.300428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:58005 (local address 127.10.71.190:42955)
0619 15:01:04.300547 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.300550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.301235 (+ 685us) server_negotiation.cc:407] Connection header received
0619 15:01:04.302464 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.302467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.302521 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.302617 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:04.303768 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.304354 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.305365 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.305591 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.305663 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.308211 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:04.308215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:04.308217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:04.308247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:04.309661 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.310124 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.310128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.310130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.310207 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.310527 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.310530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.310532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.310697 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:04.310795 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.311223 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.311368 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 15:01:04.312240 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.300728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42955 (local address 127.10.71.129:58005)
0619 15:01:04.301110 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.301143 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.302256 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.302773 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.302779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.303102 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:04.303608 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.303625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.304471 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.304474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.305265 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.305273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.305722 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.305727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.305905 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.306405 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:04.306422 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:04.308048 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:04.309775 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:04.309780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:04.309792 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:04.310026 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:04.310292 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:04.310296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:04.310298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:04.310422 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:04.310794 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:04.310800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:04.311012 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.311170 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.311497 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260619 15:01:04.313419 29928 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42955
I20260619 15:01:04.313637 29928 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:04.314244 29928 heartbeater.cc:511] Master 127.10.71.190:42955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.71.130@KRBTEST.COM; defaulting to no policy
I20260619 15:01:04.315762 29737 ts_manager.cc:194] Registered new tserver with Master: a6e90d02acbe4d528a55b77f103b7caf (127.10.71.129:43361)
I20260619 15:01:04.316852 29737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.129@KRBTEST.COM'} at 127.10.71.129:58005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
Entry for principal kudu/127.10.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab.
I20260619 15:01:04.329134 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42955
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:04.424913 29933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:04.425125 29933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:04.425200 29933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:04.428859 29933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:04.428905 29933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:04.428987 29933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:01:04.434365 29933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.130.keytab
--principal=kudu/127.10.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:04.435230 29933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:04.436112 29933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:01:04.445123 29933 server_base.cc:1034] running on GCE node
W20260619 15:01:04.445068 29938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:04.445093 29939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:04.445079 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:04.445674 29933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:04.446146 29933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:04.447302 29933 hybrid_clock.cc:648] HybridClock initialized: now 1781881264447291 us; error 25 us; skew 500 ppm
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:04.449915 29933 init.cc:377] Logged in from keytab as kudu/127.10.71.130@KRBTEST.COM (short username kudu)
I20260619 15:01:04.450668 29933 webserver.cc:460] Webserver started at http://127.10.71.130:43985/ using document root <none> and password file <none>
I20260619 15:01:04.451169 29933 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:04.451259 29933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:04.451496 29933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:04.453399 29933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "442510601f914e4c9dd43255befe4b2d"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "0c358536049e96bafa7bc12c5f3fd80f"
server_key_iv: "5ebe271dbec42ea6bde0df44139ca3d3"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.453882 29933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "442510601f914e4c9dd43255befe4b2d"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "0c358536049e96bafa7bc12c5f3fd80f"
server_key_iv: "5ebe271dbec42ea6bde0df44139ca3d3"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.457103 29933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:04.459173 29948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.460217 29933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:04.460366 29933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "442510601f914e4c9dd43255befe4b2d"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "0c358536049e96bafa7bc12c5f3fd80f"
server_key_iv: "5ebe271dbec42ea6bde0df44139ca3d3"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.460469 29933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:04.486699 29933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:04.489131 29933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:04.489303 29933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:04.489789 29933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:04.490751 29933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:04.490859 29933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.490921 29933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:04.490952 29933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.499289 29933 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:36153
I20260619 15:01:04.499315 30061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:36153 every 8 connection(s)
I20260619 15:01:04.500051 29933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:01:04.503911 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 29933
I20260619 15:01:04.504061 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 15:01:04.504379 10526 external_mini_cluster.cc:1428] Setting key 261faf1c2eb4bc90d051eb067515f225
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.130@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:04.513892 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.501682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:59567 (local address 127.10.71.190:42955)
0619 15:01:04.501835 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.501840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.502569 (+ 729us) server_negotiation.cc:407] Connection header received
0619 15:01:04.503745 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.503751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.503809 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.503908 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:04.505218 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.505857 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.507447 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.507850 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.507963 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.510297 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:04.510302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:04.510307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:04.510334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:04.511892 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.512427 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.512430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.512432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.512546 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.512859 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.512864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.512868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.513077 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:04.513173 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.513582 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.513704 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.10.71.131@KRBTEST.COM; defaulting to no policy
I20260619 15:01:04.514394 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.502024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42955 (local address 127.10.71.130:59567)
0619 15:01:04.502419 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.502457 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.503540 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.504087 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.504093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.504563 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:04.505082 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.505099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.505977 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.505981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.507323 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.507336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.507990 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.507993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.508118 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.508699 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:04.508720 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:04.510142 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:04.512008 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:04.512012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:04.512022 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:04.512251 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:04.512635 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:04.512637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:04.512638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:04.512754 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:04.513170 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:04.513174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:04.513349 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.513470 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.513664 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260619 15:01:04.515376 30062 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42955
I20260619 15:01:04.515585 30062 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:04.516057 30062 heartbeater.cc:511] Master 127.10.71.190:42955 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.131@KRBTEST.COM" created.
I20260619 15:01:04.518025 29737 ts_manager.cc:194] Registered new tserver with Master: 442510601f914e4c9dd43255befe4b2d (127.10.71.130:36153)
I20260619 15:01:04.518565 29737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.130@KRBTEST.COM'} at 127.10.71.130:59567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
Entry for principal kudu/127.10.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab.
I20260619 15:01:04.528913 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42955
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:04.623656 30067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:04.623840 30067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:04.623901 30067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:04.627274 30067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:04.627313 30067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:04.627403 30067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:01:04.631894 30067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.131.keytab
--principal=kudu/127.10.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:42955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:04.632805 30067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:04.633605 30067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:04.641283 30073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:04.641305 30075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:04.641292 30072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:04.641714 30067 server_base.cc:1034] running on GCE node
I20260619 15:01:04.642062 30067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:04.642609 30067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:04.643805 30067 hybrid_clock.cc:648] HybridClock initialized: now 1781881264643792 us; error 30 us; skew 500 ppm
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:04.646026 30067 init.cc:377] Logged in from keytab as kudu/127.10.71.131@KRBTEST.COM (short username kudu)
I20260619 15:01:04.646724 30067 webserver.cc:460] Webserver started at http://127.10.71.131:34845/ using document root <none> and password file <none>
I20260619 15:01:04.647177 30067 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:04.647259 30067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:04.647446 30067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:04.649250 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "02abeb1a8c624c1986c2758f8089550f"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "e2b47a72dc84b4b3de317ff1b83a94c4"
server_key_iv: "2f75f95cb3006039f672ea703931a8e8"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.649724 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "02abeb1a8c624c1986c2758f8089550f"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "e2b47a72dc84b4b3de317ff1b83a94c4"
server_key_iv: "2f75f95cb3006039f672ea703931a8e8"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.653116 30067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:04.655254 30082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.656394 30067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 15:01:04.656507 30067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "02abeb1a8c624c1986c2758f8089550f"
format_stamp: "Formatted at 2026-06-19 15:01:04 on dist-test-slave-3qdh"
server_key: "e2b47a72dc84b4b3de317ff1b83a94c4"
server_key_iv: "2f75f95cb3006039f672ea703931a8e8"
server_key_version: "encryptionkey@0"
I20260619 15:01:04.656603 30067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:04.672794 30067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:04.675141 30067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:04.675305 30067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:04.675768 30067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:04.676486 30067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:04.676542 30067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.676589 30067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:04.676633 30067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:04.685070 30067 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:38269
I20260619 15:01:04.685092 30195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:38269 every 8 connection(s)
I20260619 15:01:04.685830 30067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 15:01:04.693722 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30067
I20260619 15:01:04.693854 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
Jun 19 15:01:04 dist-test-slave-3qdh krb5kdc[29692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.131@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:04.694169 10526 external_mini_cluster.cc:1428] Setting key c89e5058f6ae9e99f41b55db9210beee
I20260619 15:01:04.697770 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.687364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:40259 (local address 127.10.71.190:42955)
0619 15:01:04.687491 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.687495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.688207 (+ 712us) server_negotiation.cc:407] Connection header received
0619 15:01:04.689409 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.689412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.689466 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.689571 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:04.690566 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.691027 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.692111 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.692327 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.692398 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.694698 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:04.694702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:04.694704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:04.694729 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:04.696203 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.696667 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.696669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.696671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.696738 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.697004 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.697005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.697006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.697148 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:04.697211 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.697592 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.697669 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 15:01:04.698457 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.687660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:42955 (local address 127.10.71.131:40259)
0619 15:01:04.688072 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.688108 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.689209 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.689738 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.689744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.690037 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:04.690433 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.690444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.691144 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.691147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.692012 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.692020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.692449 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.692461 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.692613 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.693042 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:04.693055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:04.694537 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:04.696339 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:04.696344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:04.696355 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:04.696590 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:04.696811 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:04.696813 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:04.696814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:04.696922 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:04.697221 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:04.697224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:04.697418 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.697550 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.697749 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260619 15:01:04.699368 30196 heartbeater.cc:347] Connected to a master server at 127.10.71.190:42955
I20260619 15:01:04.699558 30196 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:04.699931 30196 heartbeater.cc:511] Master 127.10.71.190:42955 requested a full tablet report, sending...
I20260619 15:01:04.700866 29737 ts_manager.cc:194] Registered new tserver with Master: 02abeb1a8c624c1986c2758f8089550f (127.10.71.131:38269)
I20260619 15:01:04.701359 29737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.71.131@KRBTEST.COM'} at 127.10.71.131:40259
I20260619 15:01:04.707324 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:01:04.714771 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.709051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.10.71.190:42955)
0619 15:01:04.709166 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.709175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.709301 (+ 126us) server_negotiation.cc:407] Connection header received
0619 15:01:04.709376 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.709378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.709406 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.709459 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:04.709982 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.710375 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.711017 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.711207 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.711274 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.712558 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:04.712560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:04.712562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:04.712577 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:04.713663 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.713972 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.713974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.713975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.714031 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:04.714221 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:04.714223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:04.714223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:04.714377 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:04.714461 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.714617 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.714684 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 15:01:04.716892 29737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:01:04.719561 29737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:01:04.731639 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.726620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.10.71.130:36153)
0619 15:01:04.726997 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.727007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.727029 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:01:04.727099 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.727103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.727282 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.727715 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:04.727852 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.728875 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.730092 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.731008 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.731121 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.731248 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.731411 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.731503 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":85,"threads_started":1}
I20260619 15:01:04.731632 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.726294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:36153 (local address 127.0.0.1:35428)
0619 15:01:04.726667 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.726704 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.726800 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.727428 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.727434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.727451 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:04.727573 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.727578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.729011 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.729014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.729973 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.729983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.731121 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.731124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.731220 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.731250 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.731381 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.731499 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":26,"thread_start_us":67,"threads_started":1}
I20260619 15:01:04.732575 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.726365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:38269 (local address 127.0.0.1:60648)
0619 15:01:04.727244 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.727257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.727325 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.728163 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.728166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.728182 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:04.728258 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.728262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.729398 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.729401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.730708 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.730717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.731911 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.731915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.732155 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.732216 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.732290 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.732431 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":233,"thread_start_us":86,"threads_started":1}
I20260619 15:01:04.732734 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.727049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.10.71.129:43361)
0619 15:01:04.727446 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.727457 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.727675 (+ 218us) server_negotiation.cc:407] Connection header received
0619 15:01:04.727953 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.727958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.728126 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.728247 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:04.728443 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.729229 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.730834 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.731628 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.731730 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.731847 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.732463 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.732582 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
I20260619 15:01:04.732811 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.727223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.10.71.131:38269)
0619 15:01:04.727696 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.727707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.727723 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:01:04.727781 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.727785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.727936 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.728062 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:04.728408 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.729252 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.730834 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.731628 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.731730 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.731847 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.732405 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.732507 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":105,"threads_started":1}
I20260619 15:01:04.733022 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.726884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:43361 (local address 127.0.0.1:33868)
0619 15:01:04.727483 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.727494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.727569 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.728234 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.728236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.728253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:04.728358 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.728366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.729356 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.729379 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.730707 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.730717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.731911 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.731915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.732116 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.732215 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.732288 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.732887 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":648,"thread_start_us":71,"threads_started":1}
I20260619 15:01:04.734625 29996 tablet_service.cc:1467] Processing CreateTablet for tablet bcc2a6093aa14092bf4cf38c4efa0520 (DEFAULT_TABLE table=test-table [id=d71ecc5d795e472f8ed0b930debd5e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:04.735512 29996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc2a6093aa14092bf4cf38c4efa0520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:04.735407 29862 tablet_service.cc:1467] Processing CreateTablet for tablet bcc2a6093aa14092bf4cf38c4efa0520 (DEFAULT_TABLE table=test-table [id=d71ecc5d795e472f8ed0b930debd5e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:04.735405 30130 tablet_service.cc:1467] Processing CreateTablet for tablet bcc2a6093aa14092bf4cf38c4efa0520 (DEFAULT_TABLE table=test-table [id=d71ecc5d795e472f8ed0b930debd5e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:04.736267 29862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc2a6093aa14092bf4cf38c4efa0520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:04.736267 30130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc2a6093aa14092bf4cf38c4efa0520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:04.740541 30213 tablet_bootstrap.cc:492] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Bootstrap starting.
I20260619 15:01:04.742432 30213 tablet_bootstrap.cc:654] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:04.742538 30214 tablet_bootstrap.cc:492] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: Bootstrap starting.
I20260619 15:01:04.743997 30213 log.cc:824] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:04.744530 30214 tablet_bootstrap.cc:654] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:04.744637 30215 tablet_bootstrap.cc:492] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Bootstrap starting.
I20260619 15:01:04.745724 30214 log.cc:824] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:04.745770 30213 tablet_bootstrap.cc:492] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: No bootstrap required, opened a new log
I20260619 15:01:04.745939 30213 ts_tablet_manager.cc:1397] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260619 15:01:04.746479 30215 tablet_bootstrap.cc:654] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:04.747275 30215 log.cc:824] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:04.748689 30213 raft_consensus.cc:348] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.748916 30213 raft_consensus.cc:374] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:04.748972 30213 raft_consensus.cc:729] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442510601f914e4c9dd43255befe4b2d, State: Initialized, Role: FOLLOWER
I20260619 15:01:04.748975 30214 tablet_bootstrap.cc:492] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: No bootstrap required, opened a new log
I20260619 15:01:04.749082 30215 tablet_bootstrap.cc:492] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: No bootstrap required, opened a new log
I20260619 15:01:04.749114 30214 ts_tablet_manager.cc:1397] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 15:01:04.749218 30215 ts_tablet_manager.cc:1397] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:01:04.749342 30213 consensus_queue.cc:260] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.749902 30062 heartbeater.cc:503] Master 127.10.71.190:42955 was elected leader, sending a full tablet report...
I20260619 15:01:04.750166 30213 ts_tablet_manager.cc:1428] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
W20260619 15:01:04.750888 30063 tablet.cc:2369] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:04.752108 30215 raft_consensus.cc:348] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.752272 30215 raft_consensus.cc:374] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:04.752370 30215 raft_consensus.cc:729] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02abeb1a8c624c1986c2758f8089550f, State: Initialized, Role: FOLLOWER
I20260619 15:01:04.752413 30214 raft_consensus.cc:348] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.752692 30214 raft_consensus.cc:374] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:04.752748 30214 raft_consensus.cc:729] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e90d02acbe4d528a55b77f103b7caf, State: Initialized, Role: FOLLOWER
I20260619 15:01:04.752813 30215 consensus_queue.cc:260] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.753146 30214 consensus_queue.cc:260] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.753623 30196 heartbeater.cc:503] Master 127.10.71.190:42955 was elected leader, sending a full tablet report...
I20260619 15:01:04.753681 29928 heartbeater.cc:503] Master 127.10.71.190:42955 was elected leader, sending a full tablet report...
I20260619 15:01:04.753695 30215 ts_tablet_manager.cc:1428] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 15:01:04.754992 30214 ts_tablet_manager.cc:1428] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 15:01:04.799903 29929 tablet.cc:2369] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:04.924502 30220 raft_consensus.cc:482] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:01:04.924691 30220 raft_consensus.cc:504] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.925843 30220 leader_election.cc:288] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 442510601f914e4c9dd43255befe4b2d (127.10.71.130:36153), a6e90d02acbe4d528a55b77f103b7caf (127.10.71.129:43361)
I20260619 15:01:04.930765 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.926226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:52303 (local address 127.10.71.129:43361)
0619 15:01:04.926363 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.926368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.926507 (+ 139us) server_negotiation.cc:407] Connection header received
0619 15:01:04.926567 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.926572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.926625 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.926706 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:04.926971 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.927698 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.929518 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.930124 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.930243 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.930298 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.930562 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.930631 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 15:01:04.930765 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.926097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:52101 (local address 127.10.71.130:36153)
0619 15:01:04.926210 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.926215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.926333 (+ 118us) server_negotiation.cc:407] Connection header received
0619 15:01:04.926409 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.926414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.926505 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.926600 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:04.926921 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.927699 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.929518 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.930113 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.930243 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.930298 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.930562 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.930631 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 15:01:04.930770 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.926015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:36153 (local address 127.10.71.131:52101)
0619 15:01:04.926199 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.926214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.926307 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.926625 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.926627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.926639 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:04.926779 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.926785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.927815 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.927818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.929377 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.929388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.930243 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.930247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.930416 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.930455 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.930529 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.930625 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":188}
I20260619 15:01:04.930778 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.926061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:43361 (local address 127.10.71.131:52303)
0619 15:01:04.926417 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:04.926427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:04.926501 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:04.926721 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:04.926723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:04.926737 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:04.926841 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.926849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.927815 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.927818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.929377 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:04.929388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.930243 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.930247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:04.930421 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.930461 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 15:01:04.930529 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 15:01:04.930625 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":265,"thread_start_us":107,"threads_started":1}
I20260619 15:01:04.931448 30016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc2a6093aa14092bf4cf38c4efa0520" candidate_uuid: "02abeb1a8c624c1986c2758f8089550f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442510601f914e4c9dd43255befe4b2d" is_pre_election: true
I20260619 15:01:04.931614 29882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc2a6093aa14092bf4cf38c4efa0520" candidate_uuid: "02abeb1a8c624c1986c2758f8089550f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e90d02acbe4d528a55b77f103b7caf" is_pre_election: true
I20260619 15:01:04.931730 30016 raft_consensus.cc:2454] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02abeb1a8c624c1986c2758f8089550f in term 0.
I20260619 15:01:04.931944 29882 raft_consensus.cc:2454] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02abeb1a8c624c1986c2758f8089550f in term 0.
I20260619 15:01:04.932238 30085 leader_election.cc:302] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02abeb1a8c624c1986c2758f8089550f, 442510601f914e4c9dd43255befe4b2d; no voters:
I20260619 15:01:04.932554 30220 raft_consensus.cc:2781] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:01:04.932655 30220 raft_consensus.cc:482] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:01:04.932720 30220 raft_consensus.cc:3037] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:04.933784 30220 raft_consensus.cc:504] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
I20260619 15:01:04.934149 30220 leader_election.cc:288] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [CANDIDATE]: Term 1 election: Requested vote from peers 442510601f914e4c9dd43255befe4b2d (127.10.71.130:36153), a6e90d02acbe4d528a55b77f103b7caf (127.10.71.129:43361)
I20260619 15:01:04.934442 30016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc2a6093aa14092bf4cf38c4efa0520" candidate_uuid: "02abeb1a8c624c1986c2758f8089550f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442510601f914e4c9dd43255befe4b2d"
I20260619 15:01:04.934450 29882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc2a6093aa14092bf4cf38c4efa0520" candidate_uuid: "02abeb1a8c624c1986c2758f8089550f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e90d02acbe4d528a55b77f103b7caf"
I20260619 15:01:04.934571 30016 raft_consensus.cc:3037] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:04.934588 29882 raft_consensus.cc:3037] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:04.935529 29882 raft_consensus.cc:2454] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02abeb1a8c624c1986c2758f8089550f in term 1.
I20260619 15:01:04.935612 30016 raft_consensus.cc:2454] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02abeb1a8c624c1986c2758f8089550f in term 1.
I20260619 15:01:04.935851 30086 leader_election.cc:302] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02abeb1a8c624c1986c2758f8089550f, a6e90d02acbe4d528a55b77f103b7caf; no voters:
I20260619 15:01:04.936021 30220 raft_consensus.cc:2781] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:04.936224 30220 raft_consensus.cc:686] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 1 LEADER]: Becoming Leader. State: Replica: 02abeb1a8c624c1986c2758f8089550f, State: Running, Role: LEADER
I20260619 15:01:04.936553 30220 consensus_queue.cc:237] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } }
W20260619 15:01:04.936661 30197 tablet.cc:2369] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:04.939543 29734 catalog_manager.cc:5495] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f reported cstate change: term changed from 0 to 1, leader changed from <none> to 02abeb1a8c624c1986c2758f8089550f (127.10.71.131). New cstate: current_term: 1 leader_uuid: "02abeb1a8c624c1986c2758f8089550f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02abeb1a8c624c1986c2758f8089550f" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 38269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 } health_report { overall_health: UNKNOWN } } }
I20260619 15:01:04.969178 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:04.965863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.10.71.131:38269)
0619 15:01:04.965987 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:04.965993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:04.966058 (+ 65us) server_negotiation.cc:407] Connection header received
0619 15:01:04.966226 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:04.966228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:04.966274 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:04.966381 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:04.966594 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.967018 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.967796 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:04.968018 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:04.968146 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:04.968448 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:04.968798 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:04.968923 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:04.968985 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:01:04.969033 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 15:01:04.974818 29882 raft_consensus.cc:1261] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf [term 1 FOLLOWER]: Refusing update from remote peer 02abeb1a8c624c1986c2758f8089550f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:04.974970 30016 raft_consensus.cc:1261] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 1 FOLLOWER]: Refusing update from remote peer 02abeb1a8c624c1986c2758f8089550f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:04.975459 30220 consensus_queue.cc:1035] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e90d02acbe4d528a55b77f103b7caf" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:04.975656 30223 consensus_queue.cc:1035] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [LEADER]: Connected to new peer: Peer: permanent_uuid: "442510601f914e4c9dd43255befe4b2d" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:04.984771 30227 mvcc.cc:205] Tried to move back new op lower bound from 7298585661330755584 to 7298585661182955520. Current Snapshot: MvccSnapshot[applied={T|T < 7298585661330755584 or (T in {7298585661330755584})}]
I20260619 15:01:04.985010 30226 mvcc.cc:205] Tried to move back new op lower bound from 7298585661330755584 to 7298585661182955520. Current Snapshot: MvccSnapshot[applied={T|T < 7298585661330755584}]
I20260619 15:01:04.984954 30228 mvcc.cc:205] Tried to move back new op lower bound from 7298585661330755584 to 7298585661182955520. Current Snapshot: MvccSnapshot[applied={T|T < 7298585661330755584 or (T in {7298585661330755584})}]
I20260619 15:01:04.991803 29737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:04.991954 29737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:04.994278 29737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da61939fc66149a28aee4dd5f8f0c095: Sending DeleteTablet for 3 replicas of tablet bcc2a6093aa14092bf4cf38c4efa0520
I20260619 15:01:04.995000 29996 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc2a6093aa14092bf4cf38c4efa0520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:04 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:35428
I20260619 15:01:04.995110 29862 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc2a6093aa14092bf4cf38c4efa0520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:04 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:33868
I20260619 15:01:04.995409 30130 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc2a6093aa14092bf4cf38c4efa0520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:04 UTC) from {username='kudu', principal='kudu/127.10.71.190@KRBTEST.COM'} at 127.0.0.1:60648
I20260619 15:01:04.995649 30237 tablet_replica.cc:330] T bcc2a6093aa14092bf4cf38c4efa0520 P a6e90d02acbe4d528a55b77f103b7caf: stopping tablet replica
I20260619 15:01:04.995719 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29799
I20260619 15:01:04.996268 30238 tablet_replica.cc:330] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: stopping tablet replica
I20260619 15:01:04.995777 30236 tablet_replica.cc:330] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: stopping tablet replica
I20260619 15:01:04.996690 30236 raft_consensus.cc:2229] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:01:04.996698 30238 raft_consensus.cc:2229] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:01:04.996943 30236 raft_consensus.cc:2258] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:04.997597 30238 raft_consensus.cc:2258] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:04.998322 30236 ts_tablet_manager.cc:1905] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:04.998965 30238 ts_tablet_manager.cc:1905] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:05.000819 30236 ts_tablet_manager.cc:1918] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:05.000909 30236 log.cc:1196] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/bcc2a6093aa14092bf4cf38c4efa0520
I20260619 15:01:05.001204 30236 ts_tablet_manager.cc:1939] T bcc2a6093aa14092bf4cf38c4efa0520 P 442510601f914e4c9dd43255befe4b2d: Deleting consensus metadata
I20260619 15:01:05.002395 29723 catalog_manager.cc:4841] TS 442510601f914e4c9dd43255befe4b2d (127.10.71.130:36153): tablet bcc2a6093aa14092bf4cf38c4efa0520 (table test-table [id=d71ecc5d795e472f8ed0b930debd5e36]) successfully deleted
I20260619 15:01:05.002781 30238 ts_tablet_manager.cc:1918] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:05.002924 30238 log.cc:1196] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/bcc2a6093aa14092bf4cf38c4efa0520
I20260619 15:01:05.003309 30238 ts_tablet_manager.cc:1939] T bcc2a6093aa14092bf4cf38c4efa0520 P 02abeb1a8c624c1986c2758f8089550f: Deleting consensus metadata
I20260619 15:01:05.004508 29724 catalog_manager.cc:4841] TS 02abeb1a8c624c1986c2758f8089550f (127.10.71.131:38269): tablet bcc2a6093aa14092bf4cf38c4efa0520 (table test-table [id=d71ecc5d795e472f8ed0b930debd5e36]) successfully deleted
W20260619 15:01:05.004913 29724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:43361 (error 108)
I20260619 15:01:05.005138 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29933
I20260619 15:01:05.006524 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.006258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:43361 (local address 127.0.0.1:33882)
0619 15:01:05.006384 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.006457 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 15:01:05.006726 29724 catalog_manager.cc:4584] TS a6e90d02acbe4d528a55b77f103b7caf (127.10.71.129:43361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcc2a6093aa14092bf4cf38c4efa0520: Network error: Client connection negotiation failed: client connection to 127.10.71.129:43361: connect: Connection refused (error 111)
I20260619 15:01:05.011875 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30067
I20260619 15:01:05.017992 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 29705
2026-06-19T15:01:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T15:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:05Z Disabled control of system clock
I20260619 15:01:05.045868 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39225
--webserver_interface=127.10.71.190
--webserver_port=0
--builtin_ntp_servers=127.10.71.148:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:01:05.143128 30243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:05.143317 30243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:05.143345 30243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:05.146881 30243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:05.146924 30243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:05.146939 30243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:05.146953 30243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:05.146966 30243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:05.151085 30243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:39225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:05.151882 30243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:05.153036 30243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:05.159981 30249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:05.160001 30248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:05.160032 30251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.160100 30243 server_base.cc:1034] running on GCE node
I20260619 15:01:05.160749 30243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:05.161599 30243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:05.162744 30243 hybrid_clock.cc:648] HybridClock initialized: now 1781881265162716 us; error 41 us; skew 500 ppm
I20260619 15:01:05.164147 30243 webserver.cc:460] Webserver started at http://127.10.71.190:38267/ using document root <none> and password file <none>
I20260619 15:01:05.164641 30243 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:05.164693 30243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:05.164888 30243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:05.166436 30243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "fb9db33f41df4010a3d2e87477783ed4"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "663a592bc91a580571a0d720aaf78da8"
server_key_iv: "64f7dff09d4b8d5e4bb0aba1d19c9735"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.166937 30243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "fb9db33f41df4010a3d2e87477783ed4"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "663a592bc91a580571a0d720aaf78da8"
server_key_iv: "64f7dff09d4b8d5e4bb0aba1d19c9735"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.170219 30243 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:05.172287 30257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.173498 30243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:05.173638 30243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "fb9db33f41df4010a3d2e87477783ed4"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "663a592bc91a580571a0d720aaf78da8"
server_key_iv: "64f7dff09d4b8d5e4bb0aba1d19c9735"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.173740 30243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:05.212787 30243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:05.213429 30243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:05.213652 30243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:05.220024 30243 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:39225
I20260619 15:01:05.220055 30309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:39225 every 8 connection(s)
I20260619 15:01:05.220932 30243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:05.223922 30310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:05.229254 30310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4: Bootstrap starting.
I20260619 15:01:05.230795 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30243
I20260619 15:01:05.230923 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:05.231194 10526 external_mini_cluster.cc:1428] Setting key 4c107301e330722f5b8afd0a80dda782
I20260619 15:01:05.231663 30310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:05.232481 30310 log.cc:824] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:05.234252 30310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4: No bootstrap required, opened a new log
I20260619 15:01:05.237277 30310 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9db33f41df4010a3d2e87477783ed4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39225 } }
I20260619 15:01:05.237488 30310 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:05.237571 30310 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9db33f41df4010a3d2e87477783ed4, State: Initialized, Role: FOLLOWER
I20260619 15:01:05.237936 30310 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9db33f41df4010a3d2e87477783ed4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39225 } }
I20260619 15:01:05.238068 30310 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:05.238145 30310 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:05.238235 30310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:05.239012 30310 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9db33f41df4010a3d2e87477783ed4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39225 } }
I20260619 15:01:05.239251 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.232263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.10.71.190:39225)
0619 15:01:05.232908 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.232928 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.232970 (+ 42us) server_negotiation.cc:407] Connection header received
0619 15:01:05.233551 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.233566 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.233859 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.234158 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:05.234504 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.235355 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.236155 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.236601 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.236786 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.237510 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:05.237516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:05.237530 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:05.237958 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:05.238005 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:05.238023 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:05.238115 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.238452 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.238628 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":107,"threads_started":1}
I20260619 15:01:05.239359 30310 leader_election.cc:302] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb9db33f41df4010a3d2e87477783ed4; no voters:
I20260619 15:01:05.239675 30310 leader_election.cc:288] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:05.239768 30315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:05.239949 30315 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [term 1 LEADER]: Becoming Leader. State: Replica: fb9db33f41df4010a3d2e87477783ed4, State: Running, Role: LEADER
I20260619 15:01:05.240193 30315 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9db33f41df4010a3d2e87477783ed4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39225 } }
I20260619 15:01:05.240770 30310 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:05.241245 30317 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9db33f41df4010a3d2e87477783ed4. Latest consensus state: current_term: 1 leader_uuid: "fb9db33f41df4010a3d2e87477783ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9db33f41df4010a3d2e87477783ed4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39225 } } }
I20260619 15:01:05.241361 30317 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:05.241812 30316 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb9db33f41df4010a3d2e87477783ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9db33f41df4010a3d2e87477783ed4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39225 } } }
I20260619 15:01:05.241921 30316 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:05.242942 30320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:05.245589 30320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:05.250403 30320 catalog_manager.cc:1347] Generated new cluster ID: 7720fc72ecf547d29d6187506f9a287b
I20260619 15:01:05.250480 30320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:05.255556 30320 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:05.256531 30320 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:05.263069 30320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4: Generated new TSK 0
I20260619 15:01:05.263700 30320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:05.271898 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.129:0
--local_ip_for_outbound_sockets=127.10.71.129
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39225
--builtin_ntp_servers=127.10.71.148:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 15:01:05.375720 30334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:05.375936 30334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:05.375969 30334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:05.379892 30334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:05.379936 30334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:05.380008 30334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.129
I20260619 15:01:05.384159 30334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.71.129
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.129
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:05.385005 30334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:05.386134 30334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:05.394366 30339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:05.394420 30342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.394512 30334 server_base.cc:1034] running on GCE node
W20260619 15:01:05.394469 30340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.394960 30334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:05.395493 30334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:05.396643 30334 hybrid_clock.cc:648] HybridClock initialized: now 1781881265396631 us; error 24 us; skew 500 ppm
I20260619 15:01:05.398402 30334 webserver.cc:460] Webserver started at http://127.10.71.129:41977/ using document root <none> and password file <none>
I20260619 15:01:05.398873 30334 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:05.398926 30334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:05.399148 30334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:05.400882 30334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data/instance:
uuid: "5772d9db55914357926b3a54f30e2ad5"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "34cd4f95a5a2c2b46ac9a890ce4c609b"
server_key_iv: "11d30f9861b8434796fbaf6962c3986a"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.401386 30334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance:
uuid: "5772d9db55914357926b3a54f30e2ad5"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "34cd4f95a5a2c2b46ac9a890ce4c609b"
server_key_iv: "11d30f9861b8434796fbaf6962c3986a"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.404543 30334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:05.406529 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.407626 30334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:05.407769 30334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal
uuid: "5772d9db55914357926b3a54f30e2ad5"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "34cd4f95a5a2c2b46ac9a890ce4c609b"
server_key_iv: "11d30f9861b8434796fbaf6962c3986a"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.407889 30334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:05.450313 30334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:05.450850 30334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:05.451012 30334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:05.451510 30334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:05.452405 30334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:05.452468 30334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.452528 30334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:05.452580 30334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.459854 30334 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.129:36725
I20260619 15:01:05.459882 30461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.129:36725 every 8 connection(s)
I20260619 15:01:05.460618 30334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/data/info.pb
I20260619 15:01:05.468065 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30334
I20260619 15:01:05.468178 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-0/wal/instance
I20260619 15:01:05.468178 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.462154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.129:43391 (local address 127.10.71.190:39225)
0619 15:01:05.462283 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.462286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.463126 (+ 840us) server_negotiation.cc:407] Connection header received
0619 15:01:05.464209 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.464211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.464251 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.464365 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:05.465091 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.465618 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.466532 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.466741 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.466841 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.467654 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:05.467657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:05.467658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:05.467669 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:05.467696 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:05.467703 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:05.467791 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.467952 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.468014 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 15:01:05.468487 10526 external_mini_cluster.cc:1428] Setting key 1ee765bf8f88e89e40e382bae4664ab1
I20260619 15:01:05.468886 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.462405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39225 (local address 127.10.71.129:43391)
0619 15:01:05.462972 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.463023 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.464051 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.464483 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.464488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.464560 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:05.464956 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.464966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.465713 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.465715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.466440 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.466448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.466880 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.466882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.467013 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.467457 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:01:05.467458 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:05.467489 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:01:05.467537 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:05.467818 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:05.467824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:05.467828 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.467919 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.468106 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260619 15:01:05.470007 30462 heartbeater.cc:347] Connected to a master server at 127.10.71.190:39225
I20260619 15:01:05.470207 30462 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:05.470325 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.130:0
--local_ip_for_outbound_sockets=127.10.71.130
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39225
--builtin_ntp_servers=127.10.71.148:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:01:05.470678 30462 heartbeater.cc:511] Master 127.10.71.190:39225 requested a full tablet report, sending...
I20260619 15:01:05.472213 30274 ts_manager.cc:194] Registered new tserver with Master: 5772d9db55914357926b3a54f30e2ad5 (127.10.71.129:36725)
I20260619 15:01:05.473419 30274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.129:43391
W20260619 15:01:05.565407 30465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:05.565631 30465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:05.565692 30465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:05.569366 30465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:05.569439 30465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:05.569528 30465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.130
I20260619 15:01:05.573762 30465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.71.130
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.130
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:05.574617 30465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:05.575680 30465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:05.584028 30470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:05.584043 30471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.584163 30465 server_base.cc:1034] running on GCE node
W20260619 15:01:05.584070 30473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.584600 30465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:05.585172 30465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:05.586313 30465 hybrid_clock.cc:648] HybridClock initialized: now 1781881265586286 us; error 39 us; skew 500 ppm
I20260619 15:01:05.587811 30465 webserver.cc:460] Webserver started at http://127.10.71.130:46041/ using document root <none> and password file <none>
I20260619 15:01:05.588368 30465 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:05.588428 30465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:05.588624 30465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:05.590298 30465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data/instance:
uuid: "13b69b6f7ca945e78c0d25942fca5ec9"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "19934d094720f4b93420434538ee751b"
server_key_iv: "2ea68128f62dbc4e44073cc453445d7d"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.590775 30465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance:
uuid: "13b69b6f7ca945e78c0d25942fca5ec9"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "19934d094720f4b93420434538ee751b"
server_key_iv: "2ea68128f62dbc4e44073cc453445d7d"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.594120 30465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:05.596081 30479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.597126 30465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:05.597263 30465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal
uuid: "13b69b6f7ca945e78c0d25942fca5ec9"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "19934d094720f4b93420434538ee751b"
server_key_iv: "2ea68128f62dbc4e44073cc453445d7d"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.597354 30465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:05.608378 30465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:05.608882 30465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:05.609058 30465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:05.609560 30465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:05.610344 30465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:05.610423 30465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.610493 30465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:05.610545 30465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.618438 30465 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.130:46637
I20260619 15:01:05.618456 30592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.130:46637 every 8 connection(s)
I20260619 15:01:05.619132 30465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/data/info.pb
I20260619 15:01:05.625720 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30465
I20260619 15:01:05.625839 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal/instance
I20260619 15:01:05.626150 10526 external_mini_cluster.cc:1428] Setting key 33b967236d0ade931e0a696f12c45f31
I20260619 15:01:05.627516 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.620624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:48993 (local address 127.10.71.190:39225)
0619 15:01:05.620737 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.620741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.622340 (+ 1599us) server_negotiation.cc:407] Connection header received
0619 15:01:05.623441 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.623443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.623489 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.623551 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:05.624231 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.624734 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.625745 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.626058 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.626174 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.627073 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:05.627077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:05.627079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:05.627090 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:05.627109 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:05.627117 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:05.627185 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.627341 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.627393 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 15:01:05.627909 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.71.131:0
--local_ip_for_outbound_sockets=127.10.71.131
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39225
--builtin_ntp_servers=127.10.71.148:39181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 15:01:05.628255 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.621003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39225 (local address 127.10.71.130:48993)
0619 15:01:05.622210 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.622245 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.623292 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.623677 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.623682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.623749 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:05.624127 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.624138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.624851 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.624854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.625637 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.625645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.626211 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.626216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.626364 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.626851 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:01:05.626853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:05.626884 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:01:05.626932 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:05.627196 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:05.627205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:05.627212 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.627338 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.627560 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":144,"threads_started":1}
I20260619 15:01:05.629254 30593 heartbeater.cc:347] Connected to a master server at 127.10.71.190:39225
I20260619 15:01:05.629439 30593 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:05.629846 30593 heartbeater.cc:511] Master 127.10.71.190:39225 requested a full tablet report, sending...
I20260619 15:01:05.630749 30274 ts_manager.cc:194] Registered new tserver with Master: 13b69b6f7ca945e78c0d25942fca5ec9 (127.10.71.130:46637)
I20260619 15:01:05.631531 30274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.130:48993
W20260619 15:01:05.727003 30596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:05.727207 30596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:05.727241 30596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:05.730831 30596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:05.730870 30596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:05.730918 30596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.71.131
I20260619 15:01:05.735385 30596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.71.131
--webserver_port=0
--tserver_master_addrs=127.10.71.190:39225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.71.131
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:05.736109 30596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:05.737221 30596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:05.746392 30602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.746497 30596 server_base.cc:1034] running on GCE node
W20260619 15:01:05.746402 30601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:05.746404 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:05.747037 30596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:05.747571 30596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:05.748751 30596 hybrid_clock.cc:648] HybridClock initialized: now 1781881265748734 us; error 30 us; skew 500 ppm
I20260619 15:01:05.750252 30596 webserver.cc:460] Webserver started at http://127.10.71.131:36413/ using document root <none> and password file <none>
I20260619 15:01:05.750767 30596 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:05.750852 30596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:05.751060 30596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:05.752936 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data/instance:
uuid: "c0d61123f37e440fb54898a2e94b3a42"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "ec036024e790dfe9f56723e37a63cc50"
server_key_iv: "4ad9fcd772496a7500625ef7c1b054dd"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.753407 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance:
uuid: "c0d61123f37e440fb54898a2e94b3a42"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "ec036024e790dfe9f56723e37a63cc50"
server_key_iv: "4ad9fcd772496a7500625ef7c1b054dd"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.756531 30596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:05.758554 30610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.759553 30596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:05.759668 30596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal
uuid: "c0d61123f37e440fb54898a2e94b3a42"
format_stamp: "Formatted at 2026-06-19 15:01:05 on dist-test-slave-3qdh"
server_key: "ec036024e790dfe9f56723e37a63cc50"
server_key_iv: "4ad9fcd772496a7500625ef7c1b054dd"
server_key_version: "encryptionkey@0"
I20260619 15:01:05.759763 30596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:05.782275 30596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:05.782855 30596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:05.783021 30596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:05.783494 30596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 15:01:05.784264 30596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 15:01:05.784354 30596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.784438 30596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 15:01:05.784485 30596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:05.792429 30596 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.131:42261
I20260619 15:01:05.792466 30723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.131:42261 every 8 connection(s)
I20260619 15:01:05.793251 30596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/data/info.pb
I20260619 15:01:05.801141 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.794875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.131:52467 (local address 127.10.71.190:39225)
0619 15:01:05.794998 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.795001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.795915 (+ 914us) server_negotiation.cc:407] Connection header received
0619 15:01:05.797149 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.797152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.797202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.797297 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:05.798017 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.798469 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.799489 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.799697 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.799803 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.800776 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:05.800779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:05.800781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:05.800791 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:05.800809 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:05.800816 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:05.800872 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.800973 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.801041 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 15:01:05.801936 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.795213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:39225 (local address 127.10.71.131:52467)
0619 15:01:05.795783 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.795817 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.796968 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.797441 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.797447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.797517 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:05.797899 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.797909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.798599 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.798601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.799372 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.799380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.799834 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.799837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.799979 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.800529 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 15:01:05.800531 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:05.800595 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 15:01:05.800657 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:05.800886 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:05.800892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:05.800895 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.800968 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.801168 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260619 15:01:05.802965 30724 heartbeater.cc:347] Connected to a master server at 127.10.71.190:39225
I20260619 15:01:05.803139 30724 heartbeater.cc:464] Registering TS with master...
I20260619 15:01:05.803385 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30596
I20260619 15:01:05.803485 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal/instance
I20260619 15:01:05.803660 30724 heartbeater.cc:511] Master 127.10.71.190:39225 requested a full tablet report, sending...
I20260619 15:01:05.803763 10526 external_mini_cluster.cc:1428] Setting key c6294a0ecdbaf5c3df4d09c95049e67a
I20260619 15:01:05.804636 30273 ts_manager.cc:194] Registered new tserver with Master: c0d61123f37e440fb54898a2e94b3a42 (127.10.71.131:42261)
I20260619 15:01:05.805238 30273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.71.131:52467
I20260619 15:01:05.805580 10526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 15:01:05.810199 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.807311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.10.71.190:39225)
0619 15:01:05.807404 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.807408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.807520 (+ 112us) server_negotiation.cc:407] Connection header received
0619 15:01:05.807592 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.807594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.807623 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.807687 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:05.807887 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.808374 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.809062 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.809240 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.809306 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.809853 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:05.809856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:05.809858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:05.809867 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:05.809880 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:05.809886 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:05.809934 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.809995 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.810031 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 15:01:05.812755 30273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 15:01:05.815393 30273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 15:01:05.827723 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.822571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.10.71.129:36725)
0619 15:01:05.822986 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.822998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.823440 (+ 442us) server_negotiation.cc:407] Connection header received
0619 15:01:05.823671 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.823676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.823865 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.824002 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:05.824369 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.825444 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.826638 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.827142 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.827250 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.827415 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.827494 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.827577 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":130,"threads_started":1}
I20260619 15:01:05.827718 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.822462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36725 (local address 127.0.0.1:56154)
0619 15:01:05.823177 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.823215 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.823493 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.824136 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.824139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.824151 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:05.824236 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.824244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.825567 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.825569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.826548 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.826558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.827261 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.827265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.827353 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.827383 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.827471 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.827575 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":115,"thread_start_us":79,"threads_started":1}
I20260619 15:01:05.828125 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.822593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.130:46637 (local address 127.0.0.1:59712)
0619 15:01:05.823225 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.823238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.823348 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.824242 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.824244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.824262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:05.824359 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.824365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.825281 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.825285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.826473 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.826491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.827466 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.827470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.827850 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.827879 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.827939 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.828003 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":291,"thread_start_us":89,"threads_started":1}
I20260619 15:01:05.828125 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.822892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:42261 (local address 127.0.0.1:47534)
0619 15:01:05.823514 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.823530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.823617 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.824043 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.824048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.824067 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:05.824196 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.824202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.825539 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.825542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.826507 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.826513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.827466 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.827470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.827565 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.827584 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.827631 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.827704 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":23,"thread_start_us":112,"threads_started":1}
I20260619 15:01:05.828814 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.823213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.10.71.131:42261)
0619 15:01:05.823592 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.823620 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.823641 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:01:05.823702 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.823706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.823870 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.824034 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:05.824331 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.825415 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.826616 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.827334 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.828334 (+ 1000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.828512 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.828555 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.828644 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":87,"threads_started":1}
I20260619 15:01:05.828912 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.823192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.10.71.130:46637)
0619 15:01:05.823834 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.823844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.823865 (+ 21us) server_negotiation.cc:407] Connection header received
0619 15:01:05.823933 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.823940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.824105 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.824258 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:05.824499 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.825164 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.826616 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.827334 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.827523 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.827704 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.828634 (+ 930us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.828745 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":123,"threads_started":1}
I20260619 15:01:05.830333 30396 tablet_service.cc:1467] Processing CreateTablet for tablet b39a0672d2ec4e6591fbc5ce56c7deaa (DEFAULT_TABLE table=test-table [id=fb60429d36f2439e9c8f083621baa65a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:05.831144 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39a0672d2ec4e6591fbc5ce56c7deaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:05.831139 30658 tablet_service.cc:1467] Processing CreateTablet for tablet b39a0672d2ec4e6591fbc5ce56c7deaa (DEFAULT_TABLE table=test-table [id=fb60429d36f2439e9c8f083621baa65a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:05.831139 30527 tablet_service.cc:1467] Processing CreateTablet for tablet b39a0672d2ec4e6591fbc5ce56c7deaa (DEFAULT_TABLE table=test-table [id=fb60429d36f2439e9c8f083621baa65a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 15:01:05.832086 30527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39a0672d2ec4e6591fbc5ce56c7deaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:05.832086 30658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39a0672d2ec4e6591fbc5ce56c7deaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:05.836560 30741 tablet_bootstrap.cc:492] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5: Bootstrap starting.
I20260619 15:01:05.838956 30741 tablet_bootstrap.cc:654] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:05.839534 30741 log.cc:824] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:05.840049 30742 tablet_bootstrap.cc:492] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Bootstrap starting.
I20260619 15:01:05.840545 30743 tablet_bootstrap.cc:492] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Bootstrap starting.
I20260619 15:01:05.841475 30741 tablet_bootstrap.cc:492] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5: No bootstrap required, opened a new log
I20260619 15:01:05.841665 30741 ts_tablet_manager.cc:1397] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 15:01:05.842113 30742 tablet_bootstrap.cc:654] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:05.842392 30743 tablet_bootstrap.cc:654] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:05.842900 30742 log.cc:824] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:05.843144 30743 log.cc:824] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:05.844851 30742 tablet_bootstrap.cc:492] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: No bootstrap required, opened a new log
I20260619 15:01:05.844851 30743 tablet_bootstrap.cc:492] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: No bootstrap required, opened a new log
I20260619 15:01:05.845008 30743 ts_tablet_manager.cc:1397] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:01:05.845007 30742 ts_tablet_manager.cc:1397] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 15:01:05.845170 30741 raft_consensus.cc:348] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.845359 30741 raft_consensus.cc:374] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:05.845403 30741 raft_consensus.cc:729] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5772d9db55914357926b3a54f30e2ad5, State: Initialized, Role: FOLLOWER
I20260619 15:01:05.845789 30741 consensus_queue.cc:260] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.846581 30741 ts_tablet_manager.cc:1428] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260619 15:01:05.848444 30462 heartbeater.cc:503] Master 127.10.71.190:39225 was elected leader, sending a full tablet report...
I20260619 15:01:05.848766 30742 raft_consensus.cc:348] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.848766 30743 raft_consensus.cc:348] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.849004 30742 raft_consensus.cc:374] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:05.849007 30743 raft_consensus.cc:374] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:05.849056 30743 raft_consensus.cc:729] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d61123f37e440fb54898a2e94b3a42, State: Initialized, Role: FOLLOWER
I20260619 15:01:05.849062 30742 raft_consensus.cc:729] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b69b6f7ca945e78c0d25942fca5ec9, State: Initialized, Role: FOLLOWER
I20260619 15:01:05.849432 30743 consensus_queue.cc:260] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.849432 30742 consensus_queue.cc:260] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.850139 30724 heartbeater.cc:503] Master 127.10.71.190:39225 was elected leader, sending a full tablet report...
I20260619 15:01:05.850293 30743 ts_tablet_manager.cc:1428] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 15:01:05.850548 30593 heartbeater.cc:503] Master 127.10.71.190:39225 was elected leader, sending a full tablet report...
I20260619 15:01:05.850931 30742 ts_tablet_manager.cc:1428] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 15:01:05.863591 30749 raft_consensus.cc:482] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 15:01:05.863720 30749 raft_consensus.cc:504] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.864615 30749 leader_election.cc:288] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5772d9db55914357926b3a54f30e2ad5 (127.10.71.129:36725), c0d61123f37e440fb54898a2e94b3a42 (127.10.71.131:42261)
I20260619 15:01:05.868724 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.864787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.131:42261 (local address 127.10.71.130:48049)
0619 15:01:05.864917 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.864932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.865007 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.865243 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.865246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.865256 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:05.865383 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.865388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.866222 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.866224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.867382 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.867392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.868204 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.868210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.868335 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.868460 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.868535 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.868611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":38}
I20260619 15:01:05.868743 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.864910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:38815 (local address 127.10.71.129:36725)
0619 15:01:05.865022 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.865026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.865255 (+ 229us) server_negotiation.cc:407] Connection header received
0619 15:01:05.865340 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.865341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.865371 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.865419 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:05.865654 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.866247 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.867507 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.868084 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.868192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.868233 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.868583 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.868651 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 15:01:05.868772 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.864868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.71.130:48049 (local address 127.10.71.131:42261)
0619 15:01:05.864975 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.864989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.865000 (+ 11us) server_negotiation.cc:407] Connection header received
0619 15:01:05.865121 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.865123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.865157 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.865209 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 15:01:05.865502 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.866136 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.867507 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.868080 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.868191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.868232 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.868584 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.868648 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 15:01:05.869215 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.864787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36725 (local address 127.10.71.130:38815)
0619 15:01:05.865152 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.865161 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:05.865254 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:05.865455 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:05.865460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:05.865478 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 15:01:05.865563 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.865566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.866329 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.866332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.867383 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:05.867392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.868254 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.868257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:05.868348 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.868464 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 15:01:05.869029 (+ 565us) client_negotiation.cc:240] Negotiation successful
0619 15:01:05.869094 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":184,"thread_start_us":87,"threads_started":1}
I20260619 15:01:05.869343 30678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a0672d2ec4e6591fbc5ce56c7deaa" candidate_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d61123f37e440fb54898a2e94b3a42" is_pre_election: true
I20260619 15:01:05.869604 30678 raft_consensus.cc:2454] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b69b6f7ca945e78c0d25942fca5ec9 in term 0.
I20260619 15:01:05.869762 30416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a0672d2ec4e6591fbc5ce56c7deaa" candidate_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5772d9db55914357926b3a54f30e2ad5" is_pre_election: true
W20260619 15:01:05.869948 30594 tablet.cc:2369] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:05.869983 30416 raft_consensus.cc:2454] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b69b6f7ca945e78c0d25942fca5ec9 in term 0.
I20260619 15:01:05.870115 30480 leader_election.cc:302] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b69b6f7ca945e78c0d25942fca5ec9, c0d61123f37e440fb54898a2e94b3a42; no voters:
I20260619 15:01:05.870388 30749 raft_consensus.cc:2781] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 15:01:05.870460 30749 raft_consensus.cc:482] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 15:01:05.870510 30749 raft_consensus.cc:3037] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:05.871272 30749 raft_consensus.cc:504] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.871595 30749 leader_election.cc:288] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [CANDIDATE]: Term 1 election: Requested vote from peers 5772d9db55914357926b3a54f30e2ad5 (127.10.71.129:36725), c0d61123f37e440fb54898a2e94b3a42 (127.10.71.131:42261)
I20260619 15:01:05.871840 30416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a0672d2ec4e6591fbc5ce56c7deaa" candidate_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5772d9db55914357926b3a54f30e2ad5"
I20260619 15:01:05.871870 30678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39a0672d2ec4e6591fbc5ce56c7deaa" candidate_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d61123f37e440fb54898a2e94b3a42"
I20260619 15:01:05.871956 30416 raft_consensus.cc:3037] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:05.871958 30678 raft_consensus.cc:3037] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:05.873157 30416 raft_consensus.cc:2454] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b69b6f7ca945e78c0d25942fca5ec9 in term 1.
I20260619 15:01:05.873157 30678 raft_consensus.cc:2454] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b69b6f7ca945e78c0d25942fca5ec9 in term 1.
I20260619 15:01:05.873443 30480 leader_election.cc:302] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b69b6f7ca945e78c0d25942fca5ec9, c0d61123f37e440fb54898a2e94b3a42; no voters:
I20260619 15:01:05.873610 30749 raft_consensus.cc:2781] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:05.873792 30749 raft_consensus.cc:686] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 1 LEADER]: Becoming Leader. State: Replica: 13b69b6f7ca945e78c0d25942fca5ec9, State: Running, Role: LEADER
I20260619 15:01:05.874080 30749 consensus_queue.cc:237] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } }
I20260619 15:01:05.876605 30274 catalog_manager.cc:5495] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b69b6f7ca945e78c0d25942fca5ec9 (127.10.71.130). New cstate: current_term: 1 leader_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13b69b6f7ca945e78c0d25942fca5ec9" member_type: VOTER last_known_addr { host: "127.10.71.130" port: 46637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 } health_report { overall_health: UNKNOWN } } }
I20260619 15:01:05.887499 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.884743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.10.71.130:46637)
0619 15:01:05.884836 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:05.884839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:05.884851 (+ 12us) server_negotiation.cc:407] Connection header received
0619 15:01:05.884947 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:05.884949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:05.884991 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:05.885061 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:05.885228 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.885657 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.886386 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:05.886586 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:05.886656 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:05.886874 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:05.887195 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:05.887280 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:05.887342 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:01:05.887401 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 15:01:05.891916 30678 raft_consensus.cc:1261] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 1 FOLLOWER]: Refusing update from remote peer 13b69b6f7ca945e78c0d25942fca5ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:05.891965 30416 raft_consensus.cc:1261] T b39a0672d2ec4e6591fbc5ce56c7deaa P 5772d9db55914357926b3a54f30e2ad5 [term 1 FOLLOWER]: Refusing update from remote peer 13b69b6f7ca945e78c0d25942fca5ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:05.892601 30751 consensus_queue.cc:1035] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d61123f37e440fb54898a2e94b3a42" member_type: VOTER last_known_addr { host: "127.10.71.131" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:05.892750 30749 consensus_queue.cc:1035] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5772d9db55914357926b3a54f30e2ad5" member_type: VOTER last_known_addr { host: "127.10.71.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:05.899158 30756 mvcc.cc:205] Tried to move back new op lower bound from 7298585665087635456 to 7298585665022267392. Current Snapshot: MvccSnapshot[applied={T|T < 7298585665087635456}]
I20260619 15:01:05.901391 30755 mvcc.cc:205] Tried to move back new op lower bound from 7298585665087635456 to 7298585665022267392. Current Snapshot: MvccSnapshot[applied={T|T < 7298585665087635456 or (T in {7298585665087635456})}]
I20260619 15:01:05.904229 30274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:05.904374 30274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 15:01:05.906420 30274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb9db33f41df4010a3d2e87477783ed4: Sending DeleteTablet for 3 replicas of tablet b39a0672d2ec4e6591fbc5ce56c7deaa
I20260619 15:01:05.907015 30396 tablet_service.cc:1514] Processing DeleteTablet for tablet b39a0672d2ec4e6591fbc5ce56c7deaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:05 UTC) from {username='slave'} at 127.0.0.1:56154
I20260619 15:01:05.907084 30527 tablet_service.cc:1514] Processing DeleteTablet for tablet b39a0672d2ec4e6591fbc5ce56c7deaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:05 UTC) from {username='slave'} at 127.0.0.1:59712
I20260619 15:01:05.907181 30658 tablet_service.cc:1514] Processing DeleteTablet for tablet b39a0672d2ec4e6591fbc5ce56c7deaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 15:01:05 UTC) from {username='slave'} at 127.0.0.1:47534
I20260619 15:01:05.907491 30765 tablet_replica.cc:330] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: stopping tablet replica
I20260619 15:01:05.907505 30764 tablet_replica.cc:330] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: stopping tablet replica
I20260619 15:01:05.907742 30764 raft_consensus.cc:2229] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 15:01:05.908028 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30334
I20260619 15:01:05.908079 30764 raft_consensus.cc:2258] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:05.909345 30764 ts_tablet_manager.cc:1905] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:05.910771 30765 raft_consensus.cc:2229] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 15:01:05.911041 30765 raft_consensus.cc:2258] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 15:01:05.911893 30764 ts_tablet_manager.cc:1918] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:05.911983 30764 log.cc:1196] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-1/wal/wals/b39a0672d2ec4e6591fbc5ce56c7deaa
I20260619 15:01:05.912287 30764 ts_tablet_manager.cc:1939] T b39a0672d2ec4e6591fbc5ce56c7deaa P 13b69b6f7ca945e78c0d25942fca5ec9: Deleting consensus metadata
I20260619 15:01:05.912401 30765 ts_tablet_manager.cc:1905] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 15:01:05.913291 30259 catalog_manager.cc:4841] TS 13b69b6f7ca945e78c0d25942fca5ec9 (127.10.71.130:46637): tablet b39a0672d2ec4e6591fbc5ce56c7deaa (table test-table [id=fb60429d36f2439e9c8f083621baa65a]) successfully deleted
I20260619 15:01:05.914608 30765 ts_tablet_manager.cc:1918] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 15:01:05.914729 30765 log.cc:1196] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Deleting WAL directory at /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781881028756249-10526-0/minicluster-data/ts-2/wal/wals/b39a0672d2ec4e6591fbc5ce56c7deaa
W20260619 15:01:05.914924 30260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.71.129:36725 (error 108)
I20260619 15:01:05.915069 30765 ts_tablet_manager.cc:1939] T b39a0672d2ec4e6591fbc5ce56c7deaa P c0d61123f37e440fb54898a2e94b3a42: Deleting consensus metadata
I20260619 15:01:05.915880 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30465
I20260619 15:01:05.916190 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:05.915923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.129:36725 (local address 127.0.0.1:56166)
0619 15:01:05.916016 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:05.916100 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.129:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 15:01:05.916721 30258 catalog_manager.cc:4841] TS c0d61123f37e440fb54898a2e94b3a42 (127.10.71.131:42261): tablet b39a0672d2ec4e6591fbc5ce56c7deaa (table test-table [id=fb60429d36f2439e9c8f083621baa65a]) successfully deleted
W20260619 15:01:05.917246 30260 catalog_manager.cc:4584] TS 5772d9db55914357926b3a54f30e2ad5 (127.10.71.129:36725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b39a0672d2ec4e6591fbc5ce56c7deaa: Network error: Client connection negotiation failed: client connection to 127.10.71.129:36725: connect: Connection refused (error 111)
I20260619 15:01:05.921483 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30596
I20260619 15:01:05.926421 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30243
2026-06-19T15:01:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (907 ms)
[----------] 6 tests from EncryptionPolicyTest (9814 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:05 dist-test-slave-3qdh krb5kdc[30769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:05 dist-test-slave-3qdh krb5kdc[30769](info): set up 2 sockets
Jun 19 15:01:05 dist-test-slave-3qdh krb5kdc[30769](info): commencing operation
krb5kdc: starting...
W20260619 15:01:06.876459 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:06 dist-test-slave-3qdh krb5kdc[30769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:06.981321 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:07.081291 30782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:07.081491 30782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:07.081547 30782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:07.085198 30782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:07.085239 30782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:07.085256 30782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:07.085301 30782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:07.085330 30782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:07.089872 30782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:07.090646 30782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:07.091461 30782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:07.098790 30788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:07.098857 30782 server_base.cc:1034] running on GCE node
W20260619 15:01:07.098875 30787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:07.098806 30790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:07.099537 30782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:07.100520 30782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:07.101744 30782 hybrid_clock.cc:648] HybridClock initialized: now 1781881267101716 us; error 26 us; skew 500 ppm
Jun 19 15:01:07 dist-test-slave-3qdh krb5kdc[30769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:07.109643 30782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:07.110424 30782 webserver.cc:460] Webserver started at http://127.0.0.1:40773/ using document root <none> and password file <none>
I20260619 15:01:07.110934 30782 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:07.111025 30782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:07.111223 30782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:07.113090 30782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "abf31b2846cb4489a4f97e139b5aa72c"
format_stamp: "Formatted at 2026-06-19 15:01:07 on dist-test-slave-3qdh"
server_key: "99260d167466bd9370d3d733bd56df56"
server_key_iv: "1216b0310cd01aff2c6cf3228a7fed23"
server_key_version: "encryptionkey@0"
I20260619 15:01:07.113628 30782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "abf31b2846cb4489a4f97e139b5aa72c"
format_stamp: "Formatted at 2026-06-19 15:01:07 on dist-test-slave-3qdh"
server_key: "99260d167466bd9370d3d733bd56df56"
server_key_iv: "1216b0310cd01aff2c6cf3228a7fed23"
server_key_version: "encryptionkey@0"
I20260619 15:01:07.116813 30782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 15:01:07.118954 30797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:07.120016 30782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:07.120151 30782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "abf31b2846cb4489a4f97e139b5aa72c"
format_stamp: "Formatted at 2026-06-19 15:01:07 on dist-test-slave-3qdh"
server_key: "99260d167466bd9370d3d733bd56df56"
server_key_iv: "1216b0310cd01aff2c6cf3228a7fed23"
server_key_version: "encryptionkey@0"
I20260619 15:01:07.120244 30782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:07.128929 30782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:07.131723 30782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:07.131906 30782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:07.138597 30782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33671
I20260619 15:01:07.138624 30849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33671 every 8 connection(s)
I20260619 15:01:07.139427 30782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:07.141793 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:07.146464 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30782
I20260619 15:01:07.146605 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:07.146673 30850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c: Bootstrap starting.
I20260619 15:01:07.146915 10526 external_mini_cluster.cc:1428] Setting key b30c273c5e4c97b95af9fd19977cf57c
I20260619 15:01:07.149000 30850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:07.149663 30850 log.cc:824] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:07.151270 30850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c: No bootstrap required, opened a new log
I20260619 15:01:07.154440 30850 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf31b2846cb4489a4f97e139b5aa72c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33671 } }
I20260619 15:01:07.154758 30850 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:07.154829 30850 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf31b2846cb4489a4f97e139b5aa72c, State: Initialized, Role: FOLLOWER
Jun 19 15:01:07 dist-test-slave-3qdh krb5kdc[30769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:01:07.155282 30850 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf31b2846cb4489a4f97e139b5aa72c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33671 } }
I20260619 15:01:07.155462 30850 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:07.155534 30850 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:07.155632 30850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:07.156724 30850 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf31b2846cb4489a4f97e139b5aa72c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33671 } }
I20260619 15:01:07.157070 30850 leader_election.cc:302] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf31b2846cb4489a4f97e139b5aa72c; no voters:
I20260619 15:01:07.157328 30850 leader_election.cc:288] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:07.157424 30855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:07.157644 30855 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [term 1 LEADER]: Becoming Leader. State: Replica: abf31b2846cb4489a4f97e139b5aa72c, State: Running, Role: LEADER
I20260619 15:01:07.157974 30855 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf31b2846cb4489a4f97e139b5aa72c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33671 } }
I20260619 15:01:07.158430 30850 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:07.159448 30856 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf31b2846cb4489a4f97e139b5aa72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf31b2846cb4489a4f97e139b5aa72c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33671 } } }
I20260619 15:01:07.159576 30856 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:07.159878 30857 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf31b2846cb4489a4f97e139b5aa72c. Latest consensus state: current_term: 1 leader_uuid: "abf31b2846cb4489a4f97e139b5aa72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf31b2846cb4489a4f97e139b5aa72c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33671 } } }
I20260619 15:01:07.159992 30857 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:07.161365 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:07.148176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54127 (local address 127.0.0.1:33671)
0619 15:01:07.148576 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:07.148592 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:07.148623 (+ 31us) server_negotiation.cc:407] Connection header received
0619 15:01:07.149237 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:07.149253 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:07.149546 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:07.149844 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:07.150603 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:07.151566 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:07.152461 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:07.152920 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:07.153091 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:07.155786 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:07.155792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:07.155810 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:07.155832 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:07.157699 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:07.158312 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:07.158317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:07.158323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:07.158446 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:07.158869 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:07.158875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:07.158880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:07.159392 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:07.160347 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:07.160516 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 15:01:07.160723 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":139,"threads_started":1}
W20260619 15:01:07.163060 30870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:07.163144 30870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:07.163494 30864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:07.164161 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:07.164251 30864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:07.172797 30864 catalog_manager.cc:1347] Generated new cluster ID: 2f013d7f66944449ad82218df67564e4
I20260619 15:01:07.172851 30864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:07.174427 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:07.166368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54553 (local address 127.0.0.1:33671)
0619 15:01:07.166455 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:07.166459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:07.166643 (+ 184us) server_negotiation.cc:407] Connection header received
0619 15:01:07.166776 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:07.166779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:07.166839 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:07.166915 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:07.167493 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:07.168538 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:07.169340 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:07.169625 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:07.169732 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:07.171074 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:07.171077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:07.171080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:07.171105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:07.172701 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:07.173224 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:07.173228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:07.173229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:07.173308 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:07.173620 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:07.173623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:07.173625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:07.173781 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:07.173926 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:07.174179 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 15:01:07.174265 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 15:01:07.196343 30864 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:07.197468 30864 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:07.202737 30864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf31b2846cb4489a4f97e139b5aa72c: Generated new TSK 0
I20260619 15:01:07.203341 30864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:07.209909 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30782
W20260619 15:01:07.214684 30767 connection.cc:535] client connection to 127.0.0.1:33671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33671): Connection reset by peer (error 104)
2026-06-19T15:01:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:07 dist-test-slave-3qdh krb5kdc[30882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:07 dist-test-slave-3qdh krb5kdc[30882](info): set up 2 sockets
Jun 19 15:01:07 dist-test-slave-3qdh krb5kdc[30882](info): commencing operation
krb5kdc: starting...
W20260619 15:01:08.164156 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:08 dist-test-slave-3qdh krb5kdc[30882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:08.270895 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:08.366011 30895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:08.366215 30895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:08.366248 30895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:08.370026 30895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:08.370078 30895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:08.370096 30895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:08.370138 30895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:08.370170 30895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:08.375018 30895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:08.375849 30895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:08.376698 30895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:08.384097 30901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:08.384217 30903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:08.384145 30895 server_base.cc:1034] running on GCE node
W20260619 15:01:08.384174 30900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:08.384876 30895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:08.385843 30895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:08.387035 30895 hybrid_clock.cc:648] HybridClock initialized: now 1781881268387021 us; error 27 us; skew 500 ppm
Jun 19 15:01:08 dist-test-slave-3qdh krb5kdc[30882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:08.389402 30895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:08.390098 30895 webserver.cc:460] Webserver started at http://127.0.0.1:43271/ using document root <none> and password file <none>
I20260619 15:01:08.390558 30895 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:08.390635 30895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:08.390843 30895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:08.392635 30895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "13a5ebf00d6647d5a1433481297644fd"
format_stamp: "Formatted at 2026-06-19 15:01:08 on dist-test-slave-3qdh"
server_key: "29c912f1e071fcc7e44e9f5d16aa699b"
server_key_iv: "5576b9b69512b768fea7792464e6ac1a"
server_key_version: "encryptionkey@0"
I20260619 15:01:08.393142 30895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "13a5ebf00d6647d5a1433481297644fd"
format_stamp: "Formatted at 2026-06-19 15:01:08 on dist-test-slave-3qdh"
server_key: "29c912f1e071fcc7e44e9f5d16aa699b"
server_key_iv: "5576b9b69512b768fea7792464e6ac1a"
server_key_version: "encryptionkey@0"
I20260619 15:01:08.396106 30895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:01:08.398226 30910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:08.399286 30895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:08.399430 30895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "13a5ebf00d6647d5a1433481297644fd"
format_stamp: "Formatted at 2026-06-19 15:01:08 on dist-test-slave-3qdh"
server_key: "29c912f1e071fcc7e44e9f5d16aa699b"
server_key_iv: "5576b9b69512b768fea7792464e6ac1a"
server_key_version: "encryptionkey@0"
I20260619 15:01:08.399537 30895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:08.411111 30895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:08.413658 30895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:08.413832 30895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:08.420077 30895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37415
I20260619 15:01:08.420092 30962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37415 every 8 connection(s)
I20260619 15:01:08.420917 30895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:08.423233 30963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:08.425488 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 30895
I20260619 15:01:08.425637 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:08.425905 10526 external_mini_cluster.cc:1428] Setting key 03e338dbca5bd6edce64b5773c8043b1
I20260619 15:01:08.428141 30963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd: Bootstrap starting.
I20260619 15:01:08.430264 30963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:08.430994 30963 log.cc:824] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:08.432792 30963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd: No bootstrap required, opened a new log
I20260619 15:01:08.433872 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:08.426970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58309 (local address 127.0.0.1:37415)
0619 15:01:08.427298 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:08.427311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:08.427340 (+ 29us) server_negotiation.cc:407] Connection header received
0619 15:01:08.427907 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:08.427922 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:08.428260 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:08.428534 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:08.428808 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:08.429657 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:08.430749 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:08.431256 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:08.431429 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:08.432264 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:08.432277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:08.432289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:08.432694 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:08.432757 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:08.432779 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:08.432964 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:08.433152 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 15:01:08.433279 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260619 15:01:08.436172 30963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a5ebf00d6647d5a1433481297644fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37415 } }
I20260619 15:01:08.436398 30963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:08.436496 30963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a5ebf00d6647d5a1433481297644fd, State: Initialized, Role: FOLLOWER
I20260619 15:01:08.436678 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:08.436893 30963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a5ebf00d6647d5a1433481297644fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37415 } }
I20260619 15:01:08.437016 30963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:08.437084 30963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:08.437177 30963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:08.438097 30963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a5ebf00d6647d5a1433481297644fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37415 } }
I20260619 15:01:08.438380 30963 leader_election.cc:302] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a5ebf00d6647d5a1433481297644fd; no voters:
I20260619 15:01:08.438669 30963 leader_election.cc:288] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:08.438838 30972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:08.439093 30972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [term 1 LEADER]: Becoming Leader. State: Replica: 13a5ebf00d6647d5a1433481297644fd, State: Running, Role: LEADER
I20260619 15:01:08.439445 30972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a5ebf00d6647d5a1433481297644fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37415 } }
I20260619 15:01:08.439823 30963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:08.440443 30972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a5ebf00d6647d5a1433481297644fd. Latest consensus state: current_term: 1 leader_uuid: "13a5ebf00d6647d5a1433481297644fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a5ebf00d6647d5a1433481297644fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37415 } } }
I20260619 15:01:08.440618 30972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:08.440843 30975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13a5ebf00d6647d5a1433481297644fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a5ebf00d6647d5a1433481297644fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37415 } } }
I20260619 15:01:08.440949 30982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:08.440970 30975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:08.443281 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:08.438598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59967 (local address 127.0.0.1:37415)
0619 15:01:08.438727 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:08.438731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:08.438912 (+ 181us) server_negotiation.cc:407] Connection header received
0619 15:01:08.439041 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:08.439045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:08.439090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:08.439159 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:08.439547 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:08.440220 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:08.441227 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:08.441553 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:08.441672 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:08.442678 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:08.442684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:08.442686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:08.442700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:08.442727 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:08.442740 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:08.442836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:08.443134 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 15:01:08.443163 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:01:08.443815 30982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:08.448491 30982 catalog_manager.cc:1347] Generated new cluster ID: 26ca0d1a22f142afbd6b91f1b3e8f355
I20260619 15:01:08.448557 30982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:08.474509 30982 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:08.475250 30982 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:08.487802 30982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13a5ebf00d6647d5a1433481297644fd: Generated new TSK 0
I20260619 15:01:08.488615 30982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:08.492447 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 30895
W20260619 15:01:08.497409 30880 connection.cc:535] client connection to 127.0.0.1:37415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37415): Connection reset by peer (error 104)
2026-06-19T15:01:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:08 dist-test-slave-3qdh krb5kdc[30994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:08 dist-test-slave-3qdh krb5kdc[30994](info): set up 2 sockets
Jun 19 15:01:08 dist-test-slave-3qdh krb5kdc[30994](info): commencing operation
krb5kdc: starting...
W20260619 15:01:09.448288 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:09 dist-test-slave-3qdh krb5kdc[30994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:09.553299 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:09.649400 31007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:09.649648 31007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:09.649719 31007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:09.653364 31007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:09.653447 31007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:09.653486 31007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:09.653538 31007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:09.653570 31007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:09.658192 31007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:09.658988 31007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:09.659807 31007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:09.667016 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:09.667062 31012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:09.667109 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:09.667289 31007 server_base.cc:1034] running on GCE node
I20260619 15:01:09.667847 31007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:09.668768 31007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:09.669924 31007 hybrid_clock.cc:648] HybridClock initialized: now 1781881269669907 us; error 28 us; skew 500 ppm
Jun 19 15:01:09 dist-test-slave-3qdh krb5kdc[30994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:09.672554 31007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:09.673377 31007 webserver.cc:460] Webserver started at http://127.0.0.1:45053/ using document root <none> and password file <none>
I20260619 15:01:09.673853 31007 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:09.673909 31007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:09.674141 31007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:09.675876 31007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "7e1138f28e6a41628043f7ee14cd2011"
format_stamp: "Formatted at 2026-06-19 15:01:09 on dist-test-slave-3qdh"
server_key: "282379d13b2bd4c01aa941684c66c3a1"
server_key_iv: "253edd64d3f8dc9656b70dd2c0d976a1"
server_key_version: "encryptionkey@0"
I20260619 15:01:09.676411 31007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "7e1138f28e6a41628043f7ee14cd2011"
format_stamp: "Formatted at 2026-06-19 15:01:09 on dist-test-slave-3qdh"
server_key: "282379d13b2bd4c01aa941684c66c3a1"
server_key_iv: "253edd64d3f8dc9656b70dd2c0d976a1"
server_key_version: "encryptionkey@0"
I20260619 15:01:09.679407 31007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 15:01:09.681603 31022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:09.682714 31007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:09.682847 31007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "7e1138f28e6a41628043f7ee14cd2011"
format_stamp: "Formatted at 2026-06-19 15:01:09 on dist-test-slave-3qdh"
server_key: "282379d13b2bd4c01aa941684c66c3a1"
server_key_iv: "253edd64d3f8dc9656b70dd2c0d976a1"
server_key_version: "encryptionkey@0"
I20260619 15:01:09.682960 31007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:09.686947 31007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:09.689670 31007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:09.689843 31007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:09.695804 31007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40191
I20260619 15:01:09.695827 31074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40191 every 8 connection(s)
I20260619 15:01:09.696614 31007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:09.697611 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31007
I20260619 15:01:09.697762 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:09.698081 10526 external_mini_cluster.cc:1428] Setting key 020953fb1101feea30836b42664ce98b
I20260619 15:01:09.699456 31075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:09.702714 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:09.699306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49265 (local address 127.0.0.1:40191)
0619 15:01:09.699736 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:09.699748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:09.699798 (+ 50us) server_negotiation.cc:407] Connection header received
0619 15:01:09.700419 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:09.700434 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:09.700666 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:09.700970 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:09.701686 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:09.701690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:09.701700 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:09.701710 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:09.701772 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:09.701789 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:09.701935 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:09.701999 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 15:01:09.702144 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260619 15:01:09.705271 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:09.705411 31075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011: Bootstrap starting.
I20260619 15:01:09.707456 31075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:09.708341 31075 log.cc:824] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:09.708537 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:09.707009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34507 (local address 127.0.0.1:40191)
0619 15:01:09.707136 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:09.707140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:09.707335 (+ 195us) server_negotiation.cc:407] Connection header received
0619 15:01:09.707390 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:09.707396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:09.707447 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:09.707539 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:09.708126 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:09.708131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:09.708135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:09.708148 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:09.708175 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:09.708193 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:09.708324 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:09.708386 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:01:09.708431 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 15:01:09.710050 31075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011: No bootstrap required, opened a new log
I20260619 15:01:09.713176 31075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1138f28e6a41628043f7ee14cd2011" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260619 15:01:09.713347 31075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:09.713379 31075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1138f28e6a41628043f7ee14cd2011, State: Initialized, Role: FOLLOWER
I20260619 15:01:09.713723 31075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1138f28e6a41628043f7ee14cd2011" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260619 15:01:09.713833 31075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:09.713876 31075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:09.713959 31075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:09.714644 31075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1138f28e6a41628043f7ee14cd2011" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260619 15:01:09.714905 31075 leader_election.cc:302] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e1138f28e6a41628043f7ee14cd2011; no voters:
I20260619 15:01:09.715121 31075 leader_election.cc:288] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:09.715225 31085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:09.715431 31085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [term 1 LEADER]: Becoming Leader. State: Replica: 7e1138f28e6a41628043f7ee14cd2011, State: Running, Role: LEADER
I20260619 15:01:09.715726 31085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1138f28e6a41628043f7ee14cd2011" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } }
I20260619 15:01:09.716192 31075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:09.717343 31087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e1138f28e6a41628043f7ee14cd2011. Latest consensus state: current_term: 1 leader_uuid: "7e1138f28e6a41628043f7ee14cd2011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1138f28e6a41628043f7ee14cd2011" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } } }
I20260619 15:01:09.717473 31087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:09.717705 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e1138f28e6a41628043f7ee14cd2011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1138f28e6a41628043f7ee14cd2011" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40191 } } }
I20260619 15:01:09.717792 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:09.717815 31094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:09.720551 31094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:09.725725 31094 catalog_manager.cc:1347] Generated new cluster ID: 481f47546c844d2094a6a1de0de90f48
I20260619 15:01:09.725798 31094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:09.746078 31094 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:09.746428 31094 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:09.751400 31094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e1138f28e6a41628043f7ee14cd2011: Generated new TSK 0
I20260619 15:01:09.751986 31094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:09.762063 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31007
2026-06-19T15:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:09 dist-test-slave-3qdh krb5kdc[31106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:09 dist-test-slave-3qdh krb5kdc[31106](info): set up 2 sockets
Jun 19 15:01:09 dist-test-slave-3qdh krb5kdc[31106](info): commencing operation
krb5kdc: starting...
W20260619 15:01:10.731917 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:10 dist-test-slave-3qdh krb5kdc[31106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:10.836689 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:10.933465 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:10.933676 31119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:10.933722 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:10.937129 31119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:10.937170 31119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:10.937187 31119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:10.937202 31119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:10.937215 31119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:10.941598 31119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:10.942404 31119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:10.943259 31119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:10.950718 31124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:10.950744 31125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:10.950775 31119 server_base.cc:1034] running on GCE node
W20260619 15:01:10.950789 31127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:10.951572 31119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:10.952644 31119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:10.953824 31119 hybrid_clock.cc:648] HybridClock initialized: now 1781881270953808 us; error 29 us; skew 500 ppm
Jun 19 15:01:10 dist-test-slave-3qdh krb5kdc[31106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:10.956282 31119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:10.957036 31119 webserver.cc:460] Webserver started at http://127.0.0.1:33943/ using document root <none> and password file <none>
I20260619 15:01:10.957485 31119 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:10.957571 31119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:10.957784 31119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:10.959640 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3"
format_stamp: "Formatted at 2026-06-19 15:01:10 on dist-test-slave-3qdh"
server_key: "96b0b8ec7355507a176d71cb34dbb0cc"
server_key_iv: "b5c3b686cf02611ae873dbb6b7e42d99"
server_key_version: "encryptionkey@0"
I20260619 15:01:10.960132 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3"
format_stamp: "Formatted at 2026-06-19 15:01:10 on dist-test-slave-3qdh"
server_key: "96b0b8ec7355507a176d71cb34dbb0cc"
server_key_iv: "b5c3b686cf02611ae873dbb6b7e42d99"
server_key_version: "encryptionkey@0"
I20260619 15:01:10.963316 31119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:01:10.965400 31134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:10.966567 31119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:10.966714 31119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3"
format_stamp: "Formatted at 2026-06-19 15:01:10 on dist-test-slave-3qdh"
server_key: "96b0b8ec7355507a176d71cb34dbb0cc"
server_key_iv: "b5c3b686cf02611ae873dbb6b7e42d99"
server_key_version: "encryptionkey@0"
I20260619 15:01:10.966822 31119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:10.977603 31119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:10.980381 31119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:10.980552 31119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:10.986558 31119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41947
I20260619 15:01:10.986584 31186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41947 every 8 connection(s)
I20260619 15:01:10.987301 31119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:10.989722 31187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:10.991418 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31119
I20260619 15:01:10.991554 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:10.991828 10526 external_mini_cluster.cc:1428] Setting key bc9a92c6597f7a503d475be11ef19ae6
I20260619 15:01:10.995541 31187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3: Bootstrap starting.
I20260619 15:01:10.997543 31187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:10.998178 31187 log.cc:824] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3: Log is configured to *not* fsync() on all Append() calls
Jun 19 15:01:10 dist-test-slave-3qdh krb5kdc[31106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:01:10.999778 31187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3: No bootstrap required, opened a new log
I20260619 15:01:11.002588 31187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41947 } }
I20260619 15:01:11.002846 31187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:11.002928 31187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c09e2de5dfd4ea8bb1e4f8e8823fcd3, State: Initialized, Role: FOLLOWER
I20260619 15:01:11.003396 31187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41947 } }
I20260619 15:01:11.003564 31187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:11.003635 31187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:11.003726 31187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:11.004755 31187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41947 } }
I20260619 15:01:11.005084 31187 leader_election.cc:302] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c09e2de5dfd4ea8bb1e4f8e8823fcd3; no voters:
I20260619 15:01:11.005200 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:10.993212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45283 (local address 127.0.0.1:41947)
0619 15:01:10.993609 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:10.993622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:10.993653 (+ 31us) server_negotiation.cc:407] Connection header received
0619 15:01:10.994227 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:10.994241 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:10.994528 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:10.994819 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:10.995516 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:10.996514 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:10.997276 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:10.997747 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:10.997899 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:11.000050 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:11.000054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:11.000064 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:11.000085 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:11.001982 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:11.002461 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:11.002467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:11.002475 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:11.002645 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:11.002988 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:11.002993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:11.002995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:11.003454 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:11.003654 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:11.003934 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 15:01:11.004209 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260619 15:01:11.005378 31187 leader_election.cc:288] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:11.005463 31192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:11.005683 31192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [term 1 LEADER]: Becoming Leader. State: Replica: 5c09e2de5dfd4ea8bb1e4f8e8823fcd3, State: Running, Role: LEADER
I20260619 15:01:11.005976 31192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41947 } }
I20260619 15:01:11.006490 31187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:11.007521 31194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c09e2de5dfd4ea8bb1e4f8e8823fcd3. Latest consensus state: current_term: 1 leader_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41947 } } }
I20260619 15:01:11.007647 31194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:11.007958 31193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c09e2de5dfd4ea8bb1e4f8e8823fcd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41947 } } }
I20260619 15:01:11.008052 31193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:11.008219 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:11.008837 31201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:11.011605 31201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:11.017206 31201 catalog_manager.cc:1347] Generated new cluster ID: 77a03276212c4803ae175ac2099f54e3
I20260619 15:01:11.017359 31201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:11.018461 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:11.010189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59587 (local address 127.0.0.1:41947)
0619 15:01:11.010277 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:11.010280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:11.010503 (+ 223us) server_negotiation.cc:407] Connection header received
0619 15:01:11.010619 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:11.010622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:11.010662 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:11.010734 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:11.011303 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:11.011811 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:11.012713 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:11.012969 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:11.013097 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:11.014398 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:11.014402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:11.014404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:11.014427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:11.016499 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:11.016943 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:11.016946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:11.016947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:11.017053 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:11.017373 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:11.017377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:11.017378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:11.017561 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:11.018158 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:11.018233 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 15:01:11.018335 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 15:01:11.027467 31201 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:11.028237 31201 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:11.041765 31201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c09e2de5dfd4ea8bb1e4f8e8823fcd3: Generated new TSK 0
I20260619 15:01:11.042366 31201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:11.048398 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31119
2026-06-19T15:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:11 dist-test-slave-3qdh krb5kdc[31218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:11 dist-test-slave-3qdh krb5kdc[31218](info): set up 2 sockets
Jun 19 15:01:11 dist-test-slave-3qdh krb5kdc[31218](info): commencing operation
krb5kdc: starting...
W20260619 15:01:12.000476 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:12 dist-test-slave-3qdh krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:12.104913 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:12.216833 31231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:12.217113 31231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:12.217175 31231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:12.220726 31231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:12.220791 31231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:12.220827 31231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:12.220878 31231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:12.220914 31231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:12.225740 31231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:12.226581 31231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:12.227435 31231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:12.234854 31236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:12.234951 31231 server_base.cc:1034] running on GCE node
W20260619 15:01:12.234933 31237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:12.235183 31239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:12.235698 31231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:12.236671 31231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:12.237820 31231 hybrid_clock.cc:648] HybridClock initialized: now 1781881272237799 us; error 35 us; skew 500 ppm
Jun 19 15:01:12 dist-test-slave-3qdh krb5kdc[31218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:12.240164 31231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:12.240942 31231 webserver.cc:460] Webserver started at http://127.0.0.1:36889/ using document root <none> and password file <none>
I20260619 15:01:12.241362 31231 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:12.241411 31231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:12.241545 31231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:12.243281 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "43060cf757444db9a530375837dfc35f"
format_stamp: "Formatted at 2026-06-19 15:01:12 on dist-test-slave-3qdh"
server_key: "ccbb21d9a918b2043532e3ed40d3bb8d"
server_key_iv: "88e95994b660966ee2fab9e0544ce54b"
server_key_version: "encryptionkey@0"
I20260619 15:01:12.243706 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "43060cf757444db9a530375837dfc35f"
format_stamp: "Formatted at 2026-06-19 15:01:12 on dist-test-slave-3qdh"
server_key: "ccbb21d9a918b2043532e3ed40d3bb8d"
server_key_iv: "88e95994b660966ee2fab9e0544ce54b"
server_key_version: "encryptionkey@0"
I20260619 15:01:12.246609 31231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:12.248626 31246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:12.249670 31231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:12.249789 31231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "43060cf757444db9a530375837dfc35f"
format_stamp: "Formatted at 2026-06-19 15:01:12 on dist-test-slave-3qdh"
server_key: "ccbb21d9a918b2043532e3ed40d3bb8d"
server_key_iv: "88e95994b660966ee2fab9e0544ce54b"
server_key_version: "encryptionkey@0"
I20260619 15:01:12.249879 31231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:12.270982 31231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:12.273783 31231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:12.273976 31231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:12.280136 31231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39311
I20260619 15:01:12.280160 31298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39311 every 8 connection(s)
I20260619 15:01:12.280887 31231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:12.283314 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:12.287972 31299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f: Bootstrap starting.
I20260619 15:01:12.289700 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31231
I20260619 15:01:12.289829 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:12.290107 10526 external_mini_cluster.cc:1428] Setting key e6910bf38332982e1f18c9c76af991a7
I20260619 15:01:12.290261 31299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:12.291118 31299 log.cc:824] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:12.293004 31299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f: No bootstrap required, opened a new log
I20260619 15:01:12.296686 31299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43060cf757444db9a530375837dfc35f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260619 15:01:12.296871 31299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:12.296924 31299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43060cf757444db9a530375837dfc35f, State: Initialized, Role: FOLLOWER
I20260619 15:01:12.297430 31299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43060cf757444db9a530375837dfc35f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260619 15:01:12.297590 31299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:12.297683 31299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:12.297821 31299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:12.298233 31302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:12.291119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57947 (local address 127.0.0.1:39311)
0619 15:01:12.291525 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:12.291538 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:12.291566 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:01:12.292185 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:12.292200 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:12.292572 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:12.292826 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:12.293202 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:12.294040 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:12.295170 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:12.295565 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:12.295663 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:12.296427 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:12.296431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:12.296453 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:12.296771 (+ 318us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:12.296814 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:12.296833 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:12.296950 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:12.297130 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 15:01:12.297326 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260619 15:01:12.298666 31299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43060cf757444db9a530375837dfc35f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260619 15:01:12.299005 31299 leader_election.cc:302] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43060cf757444db9a530375837dfc35f; no voters:
I20260619 15:01:12.299288 31299 leader_election.cc:288] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:12.299366 31304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:12.299548 31304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [term 1 LEADER]: Becoming Leader. State: Replica: 43060cf757444db9a530375837dfc35f, State: Running, Role: LEADER
I20260619 15:01:12.299831 31304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43060cf757444db9a530375837dfc35f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260619 15:01:12.300287 31299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:12.301028 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:12.301457 31305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43060cf757444db9a530375837dfc35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43060cf757444db9a530375837dfc35f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } } }
I20260619 15:01:12.301566 31305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:12.301759 31306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43060cf757444db9a530375837dfc35f. Latest consensus state: current_term: 1 leader_uuid: "43060cf757444db9a530375837dfc35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43060cf757444db9a530375837dfc35f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } } }
I20260619 15:01:12.301848 31306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f [sys.catalog]: This master's current role is: LEADER
W20260619 15:01:12.304996 31320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:12.305075 31320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:12.305168 31314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:12.305968 31314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:12.307339 31302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:12.303154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55407 (local address 127.0.0.1:39311)
0619 15:01:12.303286 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:12.303289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:12.303573 (+ 284us) server_negotiation.cc:407] Connection header received
0619 15:01:12.303677 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:12.303679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:12.303717 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:12.303812 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:12.304084 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:12.304813 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:12.305759 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:12.306053 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:12.306143 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:12.306887 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:12.306891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:12.306894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:12.306904 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:12.306927 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:12.306939 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:12.307069 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:12.307175 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 15:01:12.307225 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 15:01:12.310847 31314 catalog_manager.cc:1347] Generated new cluster ID: 960034cfac92413d9fd74dd1e341c0e7
I20260619 15:01:12.310902 31314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:12.322573 31314 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:12.323369 31314 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:12.331382 31314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43060cf757444db9a530375837dfc35f: Generated new TSK 0
I20260619 15:01:12.331911 31314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:12.342291 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31231
2026-06-19T15:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:12 dist-test-slave-3qdh krb5kdc[31330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:12 dist-test-slave-3qdh krb5kdc[31330](info): set up 2 sockets
Jun 19 15:01:12 dist-test-slave-3qdh krb5kdc[31330](info): commencing operation
krb5kdc: starting...
W20260619 15:01:13.308203 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:13 dist-test-slave-3qdh krb5kdc[31330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:13.411705 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:13.511041 31343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:13.511241 31343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:13.511276 31343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:13.515020 31343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:13.515066 31343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:13.515084 31343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:13.515097 31343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:13.515111 31343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:13.519507 31343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:13.520336 31343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:13.521178 31343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:13.528788 31349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:13.528864 31348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:13.528788 31351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:13.528904 31343 server_base.cc:1034] running on GCE node
I20260619 15:01:13.529557 31343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:13.530498 31343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:13.531678 31343 hybrid_clock.cc:648] HybridClock initialized: now 1781881273531665 us; error 31 us; skew 500 ppm
Jun 19 15:01:13 dist-test-slave-3qdh krb5kdc[31330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:13.534217 31343 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:13.534956 31343 webserver.cc:460] Webserver started at http://127.0.0.1:35835/ using document root <none> and password file <none>
I20260619 15:01:13.535404 31343 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:13.535499 31343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:13.535683 31343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:13.537528 31343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "f0fd4f849409411ab294fbb89e7e8df0"
format_stamp: "Formatted at 2026-06-19 15:01:13 on dist-test-slave-3qdh"
server_key: "7074ff552266716b0c6c1961a2508715"
server_key_iv: "4cc915c7ef4b00592c98afc9c38f8fa5"
server_key_version: "encryptionkey@0"
I20260619 15:01:13.538036 31343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "f0fd4f849409411ab294fbb89e7e8df0"
format_stamp: "Formatted at 2026-06-19 15:01:13 on dist-test-slave-3qdh"
server_key: "7074ff552266716b0c6c1961a2508715"
server_key_iv: "4cc915c7ef4b00592c98afc9c38f8fa5"
server_key_version: "encryptionkey@0"
I20260619 15:01:13.541230 31343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 15:01:13.543363 31358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:13.544620 31343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 15:01:13.544754 31343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "f0fd4f849409411ab294fbb89e7e8df0"
format_stamp: "Formatted at 2026-06-19 15:01:13 on dist-test-slave-3qdh"
server_key: "7074ff552266716b0c6c1961a2508715"
server_key_iv: "4cc915c7ef4b00592c98afc9c38f8fa5"
server_key_version: "encryptionkey@0"
I20260619 15:01:13.544871 31343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:13.549069 31343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:13.551578 31343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:13.551757 31343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:13.558279 31343 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37397
I20260619 15:01:13.558290 31410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37397 every 8 connection(s)
I20260619 15:01:13.559055 31343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:13.561453 31411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:13.566212 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31343
I20260619 15:01:13.566270 31411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0: Bootstrap starting.
I20260619 15:01:13.566408 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:13.566682 10526 external_mini_cluster.cc:1428] Setting key 5a5ed57f084c5b412646334b887aad3f
I20260619 15:01:13.568872 31411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:13.569686 31411 log.cc:824] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:13.571841 31411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0: No bootstrap required, opened a new log
I20260619 15:01:13.571947 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:13.567876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47139 (local address 127.0.0.1:37397)
0619 15:01:13.568330 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:13.568347 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:13.568386 (+ 39us) server_negotiation.cc:407] Connection header received
0619 15:01:13.569212 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:13.569233 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:13.569560 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:13.569882 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:13.570593 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:13.570598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:13.570628 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:13.570639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:13.570693 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:13.570716 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:13.570817 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:13.570979 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 15:01:13.571159 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":105,"threads_started":1}
I20260619 15:01:13.574496 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:13.575837 31411 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd4f849409411ab294fbb89e7e8df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37397 } }
I20260619 15:01:13.576025 31411 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:13.576097 31411 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fd4f849409411ab294fbb89e7e8df0, State: Initialized, Role: FOLLOWER
I20260619 15:01:13.576577 31411 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd4f849409411ab294fbb89e7e8df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37397 } }
I20260619 15:01:13.576742 31411 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:13.576812 31411 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:13.576924 31411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:13.578071 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:13.576471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56559 (local address 127.0.0.1:37397)
0619 15:01:13.576591 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:13.576596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:13.576882 (+ 286us) server_negotiation.cc:407] Connection header received
0619 15:01:13.577032 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:13.577035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:13.577064 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:13.577134 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:13.577703 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:13.577705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:13.577707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:13.577715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:13.577734 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:13.577743 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:13.577811 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:13.577933 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 15:01:13.577965 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 15:01:13.578115 31411 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd4f849409411ab294fbb89e7e8df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37397 } }
I20260619 15:01:13.578466 31411 leader_election.cc:302] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0fd4f849409411ab294fbb89e7e8df0; no voters:
I20260619 15:01:13.578773 31411 leader_election.cc:288] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:13.578958 31421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:13.579140 31421 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [term 1 LEADER]: Becoming Leader. State: Replica: f0fd4f849409411ab294fbb89e7e8df0, State: Running, Role: LEADER
I20260619 15:01:13.579443 31421 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd4f849409411ab294fbb89e7e8df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37397 } }
I20260619 15:01:13.579924 31411 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:13.580909 31422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0fd4f849409411ab294fbb89e7e8df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd4f849409411ab294fbb89e7e8df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37397 } } }
I20260619 15:01:13.581037 31422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:13.581382 31423 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0fd4f849409411ab294fbb89e7e8df0. Latest consensus state: current_term: 1 leader_uuid: "f0fd4f849409411ab294fbb89e7e8df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fd4f849409411ab294fbb89e7e8df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37397 } } }
I20260619 15:01:13.581452 31430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:13.581492 31423 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:13.584477 31430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:13.589255 31430 catalog_manager.cc:1347] Generated new cluster ID: 863e56fe964a44ddaf0c7a8d8f351897
I20260619 15:01:13.589308 31430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:13.628374 31430 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:13.628659 31430 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:13.634523 31430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0fd4f849409411ab294fbb89e7e8df0: Generated new TSK 0
I20260619 15:01:13.635157 31430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:13.643679 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31343
2026-06-19T15:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:13 dist-test-slave-3qdh krb5kdc[31442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:13 dist-test-slave-3qdh krb5kdc[31442](info): set up 2 sockets
Jun 19 15:01:13 dist-test-slave-3qdh krb5kdc[31442](info): commencing operation
krb5kdc: starting...
W20260619 15:01:14.592139 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:14 dist-test-slave-3qdh krb5kdc[31442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:14.693414 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:14.786955 31455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:14.787153 31455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:14.787197 31455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:14.790668 31455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:14.790712 31455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:14.790729 31455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:14.790743 31455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:14.790757 31455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:14.795320 31455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:14.796049 31455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:14.796810 31455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:14.804001 31461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:14.804009 31463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:14.804064 31455 server_base.cc:1034] running on GCE node
W20260619 15:01:14.804004 31460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:14.804720 31455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:14.805619 31455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:14.806797 31455 hybrid_clock.cc:648] HybridClock initialized: now 1781881274806784 us; error 28 us; skew 500 ppm
Jun 19 15:01:14 dist-test-slave-3qdh krb5kdc[31442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:14.809360 31455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:14.809988 31455 webserver.cc:460] Webserver started at http://127.0.0.1:39487/ using document root <none> and password file <none>
I20260619 15:01:14.810575 31455 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:14.810654 31455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:14.810849 31455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:14.813393 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "6ea046dd4585444f9aed3b370ce39b4c"
format_stamp: "Formatted at 2026-06-19 15:01:14 on dist-test-slave-3qdh"
server_key: "3a30608a4d8c436f024160835cf885f4"
server_key_iv: "49b338da505947cf67aaedc8cb7bc28d"
server_key_version: "encryptionkey@0"
I20260619 15:01:14.814054 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "6ea046dd4585444f9aed3b370ce39b4c"
format_stamp: "Formatted at 2026-06-19 15:01:14 on dist-test-slave-3qdh"
server_key: "3a30608a4d8c436f024160835cf885f4"
server_key_iv: "49b338da505947cf67aaedc8cb7bc28d"
server_key_version: "encryptionkey@0"
I20260619 15:01:14.818249 31455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 15:01:14.820690 31470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:14.821658 31455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:01:14.821776 31455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "6ea046dd4585444f9aed3b370ce39b4c"
format_stamp: "Formatted at 2026-06-19 15:01:14 on dist-test-slave-3qdh"
server_key: "3a30608a4d8c436f024160835cf885f4"
server_key_iv: "49b338da505947cf67aaedc8cb7bc28d"
server_key_version: "encryptionkey@0"
I20260619 15:01:14.821867 31455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:14.833676 31455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:14.835887 31455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:14.836104 31455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:14.842243 31455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36553
I20260619 15:01:14.842265 31522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36553 every 8 connection(s)
I20260619 15:01:14.843012 31455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:14.845419 31523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:14.848027 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31455
I20260619 15:01:14.848156 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:14.848464 10526 external_mini_cluster.cc:1428] Setting key 101a4aa067a66945286b4aa976d2afde
I20260619 15:01:14.850549 31523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c: Bootstrap starting.
I20260619 15:01:14.853003 31523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:14.853820 31523 log.cc:824] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:14.855674 31523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c: No bootstrap required, opened a new log
Jun 19 15:01:14 dist-test-slave-3qdh krb5kdc[31442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:01:14.859467 31523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea046dd4585444f9aed3b370ce39b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260619 15:01:14.859727 31523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:14.859793 31523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea046dd4585444f9aed3b370ce39b4c, State: Initialized, Role: FOLLOWER
I20260619 15:01:14.860232 31523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea046dd4585444f9aed3b370ce39b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260619 15:01:14.860427 31523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:14.860500 31523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:14.860590 31523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:14.861529 31523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea046dd4585444f9aed3b370ce39b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260619 15:01:14.861862 31523 leader_election.cc:302] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ea046dd4585444f9aed3b370ce39b4c; no voters:
I20260619 15:01:14.862169 31523 leader_election.cc:288] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:14.862286 31528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:14.862362 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:14.849700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:40243 (local address 127.0.0.1:36553)
0619 15:01:14.850063 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:14.850079 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:14.850116 (+ 37us) server_negotiation.cc:407] Connection header received
0619 15:01:14.850852 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:14.850871 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:14.851267 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:14.851593 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:14.852217 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:14.853677 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:14.854557 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:14.855193 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:14.855337 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:14.857556 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:14.857561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:14.857575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:14.857607 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:14.859410 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:14.859896 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:14.859901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:14.859907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:14.860041 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:14.860374 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:14.860380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:14.860383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:14.860838 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:14.861061 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:14.861251 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 15:01:14.861448 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20260619 15:01:14.862563 31528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [term 1 LEADER]: Becoming Leader. State: Replica: 6ea046dd4585444f9aed3b370ce39b4c, State: Running, Role: LEADER
I20260619 15:01:14.862830 31528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea046dd4585444f9aed3b370ce39b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260619 15:01:14.863283 31523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:14.863938 31529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ea046dd4585444f9aed3b370ce39b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea046dd4585444f9aed3b370ce39b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260619 15:01:14.864044 31529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:14.864253 31529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ea046dd4585444f9aed3b370ce39b4c. Latest consensus state: current_term: 1 leader_uuid: "6ea046dd4585444f9aed3b370ce39b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea046dd4585444f9aed3b370ce39b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260619 15:01:14.864494 31529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:14.864832 31536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:14.865193 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:14.867686 31536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:14.872267 31536 catalog_manager.cc:1347] Generated new cluster ID: 1279a24427734dc1ba05b9fe4cd8db3a
I20260619 15:01:14.872323 31536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:14.875126 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:14.867082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:38657 (local address 127.0.0.1:36553)
0619 15:01:14.867166 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:14.867170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:14.867341 (+ 171us) server_negotiation.cc:407] Connection header received
0619 15:01:14.867462 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:14.867464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:14.867509 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:14.867584 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:14.868524 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:14.869197 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:14.870275 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:14.870595 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:14.870660 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:14.872128 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:14.872131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:14.872133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:14.872153 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:14.873565 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:14.874001 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:14.874004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:14.874007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:14.874080 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:14.874372 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:14.874375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:14.874376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:14.874513 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:14.874650 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:14.874886 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 15:01:14.874966 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 15:01:14.898553 31536 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:14.899250 31536 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:14.903997 31536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ea046dd4585444f9aed3b370ce39b4c: Generated new TSK 0
I20260619 15:01:14.904649 31536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:14.916100 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31455
W20260619 15:01:14.921904 31440 connection.cc:535] client connection to 127.0.0.1:36553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36553): Connection reset by peer (error 104)
2026-06-19T15:01:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:14 dist-test-slave-3qdh krb5kdc[31554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:14 dist-test-slave-3qdh krb5kdc[31554](info): set up 2 sockets
Jun 19 15:01:14 dist-test-slave-3qdh krb5kdc[31554](info): commencing operation
krb5kdc: starting...
W20260619 15:01:15.876607 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:15 dist-test-slave-3qdh krb5kdc[31554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:15.989090 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:16.094806 31567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:16.095095 31567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:16.095172 31567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:16.099051 31567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:16.099138 31567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:16.099185 31567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:16.099241 31567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:16.099282 31567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:16.104218 31567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:16.105177 31567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:16.106120 31567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:16.114228 31572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:16.114239 31575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:16.114239 31573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:16.114365 31567 server_base.cc:1034] running on GCE node
I20260619 15:01:16.115023 31567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:16.115976 31567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:16.117161 31567 hybrid_clock.cc:648] HybridClock initialized: now 1781881276117119 us; error 55 us; skew 500 ppm
Jun 19 15:01:16 dist-test-slave-3qdh krb5kdc[31554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:16.119863 31567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:16.120718 31567 webserver.cc:460] Webserver started at http://127.0.0.1:43895/ using document root <none> and password file <none>
I20260619 15:01:16.121227 31567 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:16.121311 31567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:16.121534 31567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:16.123426 31567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "146f089ced08404582e73db05071e13c"
format_stamp: "Formatted at 2026-06-19 15:01:16 on dist-test-slave-3qdh"
server_key: "b8f18de74908ea12c20e28937bfcfe10"
server_key_iv: "027b5724bdd813279cd83baa56b25aa8"
server_key_version: "encryptionkey@0"
I20260619 15:01:16.123955 31567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "146f089ced08404582e73db05071e13c"
format_stamp: "Formatted at 2026-06-19 15:01:16 on dist-test-slave-3qdh"
server_key: "b8f18de74908ea12c20e28937bfcfe10"
server_key_iv: "027b5724bdd813279cd83baa56b25aa8"
server_key_version: "encryptionkey@0"
I20260619 15:01:16.127365 31567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 15:01:16.129585 31582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:16.130713 31567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:16.130851 31567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "146f089ced08404582e73db05071e13c"
format_stamp: "Formatted at 2026-06-19 15:01:16 on dist-test-slave-3qdh"
server_key: "b8f18de74908ea12c20e28937bfcfe10"
server_key_iv: "027b5724bdd813279cd83baa56b25aa8"
server_key_version: "encryptionkey@0"
I20260619 15:01:16.130954 31567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:16.149544 31567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:16.152500 31567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:16.152695 31567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:16.159926 31567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32859
I20260619 15:01:16.159953 31634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32859 every 8 connection(s)
I20260619 15:01:16.160794 31567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:16.163210 31635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:16.163908 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31567
I20260619 15:01:16.164049 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:16.164350 10526 external_mini_cluster.cc:1428] Setting key 92dba7cd6322c038e82402b951d6d43a
I20260619 15:01:16.168808 31635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c: Bootstrap starting.
I20260619 15:01:16.170910 31635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:16.171814 31635 log.cc:824] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:16.173031 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:16.165644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:59131 (local address 127.0.0.1:32859)
0619 15:01:16.166065 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:16.166079 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:16.166111 (+ 32us) server_negotiation.cc:407] Connection header received
0619 15:01:16.166713 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:16.166728 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:16.167046 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:16.167354 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:16.167728 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:16.168720 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:16.169665 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:16.170119 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:16.170287 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:16.171221 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:16.171227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:16.171246 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:16.171734 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:16.171796 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:16.171829 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:16.171972 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:16.172172 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 15:01:16.172325 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260619 15:01:16.173764 31635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c: No bootstrap required, opened a new log
I20260619 15:01:16.175668 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:16.177465 31635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146f089ced08404582e73db05071e13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 15:01:16.177723 31635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:16.177786 31635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146f089ced08404582e73db05071e13c, State: Initialized, Role: FOLLOWER
I20260619 15:01:16.178246 31635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146f089ced08404582e73db05071e13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 15:01:16.178409 31635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:16.178498 31635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:16.178596 31635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:16.179533 31635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146f089ced08404582e73db05071e13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 15:01:16.179854 31635 leader_election.cc:302] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 146f089ced08404582e73db05071e13c; no voters:
I20260619 15:01:16.180168 31635 leader_election.cc:288] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:16.180250 31645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:16.180455 31645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [term 1 LEADER]: Becoming Leader. State: Replica: 146f089ced08404582e73db05071e13c, State: Running, Role: LEADER
I20260619 15:01:16.180792 31645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146f089ced08404582e73db05071e13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 15:01:16.181524 31635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:16.182420 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:16.177591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:39023 (local address 127.0.0.1:32859)
0619 15:01:16.177694 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:16.177700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:16.177854 (+ 154us) server_negotiation.cc:407] Connection header received
0619 15:01:16.178054 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:16.178058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:16.178100 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:16.178166 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:16.178483 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:16.179071 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:16.180182 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:16.180619 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:16.180725 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:16.181618 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:16.181626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:16.181628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:16.181643 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:16.181672 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:16.181684 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:16.181813 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:16.182232 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 15:01:16.182272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:01:16.182549 31646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "146f089ced08404582e73db05071e13c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146f089ced08404582e73db05071e13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } } }
I20260619 15:01:16.182833 31646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:16.182736 31647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 146f089ced08404582e73db05071e13c. Latest consensus state: current_term: 1 leader_uuid: "146f089ced08404582e73db05071e13c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146f089ced08404582e73db05071e13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } } }
I20260619 15:01:16.183135 31647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:16.183434 31654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:16.186378 31654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:16.191220 31654 catalog_manager.cc:1347] Generated new cluster ID: 0a449f01a05940d7aa1874f99510f32e
I20260619 15:01:16.191280 31654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:16.205286 31654 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:16.206341 31654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:16.211642 31654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 146f089ced08404582e73db05071e13c: Generated new TSK 0
I20260619 15:01:16.212333 31654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:16.221141 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31567
W20260619 15:01:16.225728 31552 connection.cc:535] client connection to 127.0.0.1:32859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32859): Connection reset by peer (error 104)
2026-06-19T15:01:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:16 dist-test-slave-3qdh krb5kdc[31666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:16 dist-test-slave-3qdh krb5kdc[31666](info): set up 2 sockets
Jun 19 15:01:16 dist-test-slave-3qdh krb5kdc[31666](info): commencing operation
krb5kdc: starting...
W20260619 15:01:17.168036 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:17 dist-test-slave-3qdh krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:17.271097 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:17.364549 31679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:17.364738 31679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:17.364773 31679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:17.368201 31679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:17.368248 31679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:17.368268 31679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:17.368347 31679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:17.368391 31679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:17.372895 31679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:17.373910 31679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:17.374708 31679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:17.381789 31685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:17.381819 31684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:17.381793 31687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:17.382005 31679 server_base.cc:1034] running on GCE node
I20260619 15:01:17.382556 31679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:17.383455 31679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:17.384649 31679 hybrid_clock.cc:648] HybridClock initialized: now 1781881277384635 us; error 29 us; skew 500 ppm
Jun 19 15:01:17 dist-test-slave-3qdh krb5kdc[31666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:17.391757 31679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:17.392519 31679 webserver.cc:460] Webserver started at http://127.0.0.1:38759/ using document root <none> and password file <none>
I20260619 15:01:17.393000 31679 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:17.393088 31679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:17.393290 31679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:17.394994 31679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "54646a651dc24801ada40b4e6d16e955"
format_stamp: "Formatted at 2026-06-19 15:01:17 on dist-test-slave-3qdh"
server_key: "caa52499c8650c949a75b14002764efb"
server_key_iv: "fef96a41423ec5aa034c07676d39cf93"
server_key_version: "encryptionkey@0"
I20260619 15:01:17.395466 31679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "54646a651dc24801ada40b4e6d16e955"
format_stamp: "Formatted at 2026-06-19 15:01:17 on dist-test-slave-3qdh"
server_key: "caa52499c8650c949a75b14002764efb"
server_key_iv: "fef96a41423ec5aa034c07676d39cf93"
server_key_version: "encryptionkey@0"
I20260619 15:01:17.398661 31679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 15:01:17.400578 31694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:17.401536 31679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 15:01:17.401650 31679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "54646a651dc24801ada40b4e6d16e955"
format_stamp: "Formatted at 2026-06-19 15:01:17 on dist-test-slave-3qdh"
server_key: "caa52499c8650c949a75b14002764efb"
server_key_iv: "fef96a41423ec5aa034c07676d39cf93"
server_key_version: "encryptionkey@0"
I20260619 15:01:17.401739 31679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:17.405638 31679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:17.408035 31679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:17.408216 31679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:17.414084 31679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45121
I20260619 15:01:17.414094 31746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45121 every 8 connection(s)
I20260619 15:01:17.414860 31679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:17.415506 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31679
I20260619 15:01:17.415627 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:17.415879 10526 external_mini_cluster.cc:1428] Setting key e08f0eb3e24f26beb05f9b6a285c64d1
I20260619 15:01:17.417650 31747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:17.420239 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:17.417036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:52853 (local address 127.0.0.1:45121)
0619 15:01:17.417375 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:17.417395 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:17.417425 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:01:17.418067 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:17.418082 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:17.418308 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:17.418556 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:17.419296 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:17.419299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:17.419309 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:17.419317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:17.419357 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:17.419372 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:17.419459 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:17.419567 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 15:01:17.419700 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260619 15:01:17.422408 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:17.422869 31747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955: Bootstrap starting.
I20260619 15:01:17.425299 31747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:17.425992 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:17.424520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:49817 (local address 127.0.0.1:45121)
0619 15:01:17.424620 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:17.424624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:17.424789 (+ 165us) server_negotiation.cc:407] Connection header received
0619 15:01:17.424885 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:17.424887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:17.424916 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:17.424986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:17.425635 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:17.425638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:17.425639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:17.425648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:17.425668 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:17.425676 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:17.425738 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:17.425839 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 15:01:17.425866 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 15:01:17.426115 31747 log.cc:824] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:17.427632 31747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955: No bootstrap required, opened a new log
I20260619 15:01:17.430490 31747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54646a651dc24801ada40b4e6d16e955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260619 15:01:17.430653 31747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:17.430681 31747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54646a651dc24801ada40b4e6d16e955, State: Initialized, Role: FOLLOWER
I20260619 15:01:17.430962 31747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54646a651dc24801ada40b4e6d16e955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260619 15:01:17.431113 31747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:17.431171 31747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:17.431260 31747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:17.431967 31747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54646a651dc24801ada40b4e6d16e955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260619 15:01:17.432263 31747 leader_election.cc:302] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54646a651dc24801ada40b4e6d16e955; no voters:
I20260619 15:01:17.432586 31747 leader_election.cc:288] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:17.432687 31757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:17.432934 31757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [term 1 LEADER]: Becoming Leader. State: Replica: 54646a651dc24801ada40b4e6d16e955, State: Running, Role: LEADER
I20260619 15:01:17.433236 31757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54646a651dc24801ada40b4e6d16e955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260619 15:01:17.433738 31747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:17.434770 31759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54646a651dc24801ada40b4e6d16e955. Latest consensus state: current_term: 1 leader_uuid: "54646a651dc24801ada40b4e6d16e955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54646a651dc24801ada40b4e6d16e955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } } }
I20260619 15:01:17.434880 31759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:17.435252 31766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:17.435417 31758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54646a651dc24801ada40b4e6d16e955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54646a651dc24801ada40b4e6d16e955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } } }
I20260619 15:01:17.435557 31758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:17.437402 31766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:17.441550 31766 catalog_manager.cc:1347] Generated new cluster ID: a21d7bb507944d5e9b8cca36129f82ba
I20260619 15:01:17.441601 31766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:17.454730 31766 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:17.454998 31766 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:17.459014 31766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54646a651dc24801ada40b4e6d16e955: Generated new TSK 0
I20260619 15:01:17.459497 31766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:17.472174 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31679
2026-06-19T15:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:17 dist-test-slave-3qdh krb5kdc[31778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:17 dist-test-slave-3qdh krb5kdc[31778](info): set up 2 sockets
Jun 19 15:01:17 dist-test-slave-3qdh krb5kdc[31778](info): commencing operation
krb5kdc: starting...
W20260619 15:01:18.416376 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:18 dist-test-slave-3qdh krb5kdc[31778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:18.518638 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:18.613569 31791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:18.613777 31791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:18.613811 31791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:18.617251 31791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:18.617293 31791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:18.617344 31791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:18.617419 31791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:18.617452 31791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:18.621914 31791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:18.622666 31791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:18.623426 31791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:18.630366 31796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:18.630400 31791 server_base.cc:1034] running on GCE node
W20260619 15:01:18.630383 31797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:18.630373 31799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:18.631080 31791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:18.631925 31791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:18.633176 31791 hybrid_clock.cc:648] HybridClock initialized: now 1781881278633158 us; error 26 us; skew 500 ppm
Jun 19 15:01:18 dist-test-slave-3qdh krb5kdc[31778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:18.635650 31791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:18.636430 31791 webserver.cc:460] Webserver started at http://127.0.0.1:38815/ using document root <none> and password file <none>
I20260619 15:01:18.636871 31791 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:18.636965 31791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:18.637161 31791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:18.638813 31791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "c7625d5c62444d3aa409dd2c4c0ac778"
format_stamp: "Formatted at 2026-06-19 15:01:18 on dist-test-slave-3qdh"
server_key: "d4e4de4b83060f1ea07f78bc8003d3b4"
server_key_iv: "d1e16a423665d9a1d94815406ede62dd"
server_key_version: "encryptionkey@0"
I20260619 15:01:18.639321 31791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "c7625d5c62444d3aa409dd2c4c0ac778"
format_stamp: "Formatted at 2026-06-19 15:01:18 on dist-test-slave-3qdh"
server_key: "d4e4de4b83060f1ea07f78bc8003d3b4"
server_key_iv: "d1e16a423665d9a1d94815406ede62dd"
server_key_version: "encryptionkey@0"
I20260619 15:01:18.642350 31791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 15:01:18.644490 31806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:18.645443 31791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:18.645577 31791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "c7625d5c62444d3aa409dd2c4c0ac778"
format_stamp: "Formatted at 2026-06-19 15:01:18 on dist-test-slave-3qdh"
server_key: "d4e4de4b83060f1ea07f78bc8003d3b4"
server_key_iv: "d1e16a423665d9a1d94815406ede62dd"
server_key_version: "encryptionkey@0"
I20260619 15:01:18.645673 31791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:18.668547 31791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:18.671327 31791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:18.671509 31791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:18.677662 31791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37681
I20260619 15:01:18.677672 31858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37681 every 8 connection(s)
I20260619 15:01:18.678454 31791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:18.680677 31859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:18.683202 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31791
I20260619 15:01:18.683334 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:18.683640 10526 external_mini_cluster.cc:1428] Setting key fecef461a92c25348a555296aa29f99e
I20260619 15:01:18.685603 31859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778: Bootstrap starting.
I20260619 15:01:18.687866 31859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:18.688654 31859 log.cc:824] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:18.690260 31859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778: No bootstrap required, opened a new log
Jun 19 15:01:18 dist-test-slave-3qdh krb5kdc[31778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 15:01:18.693513 31859 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37681 } }
I20260619 15:01:18.693739 31859 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:18.693804 31859 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7625d5c62444d3aa409dd2c4c0ac778, State: Initialized, Role: FOLLOWER
I20260619 15:01:18.694219 31859 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37681 } }
I20260619 15:01:18.694362 31859 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:18.694428 31859 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:18.694542 31859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:18.695497 31859 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37681 } }
I20260619 15:01:18.695801 31859 leader_election.cc:302] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7625d5c62444d3aa409dd2c4c0ac778; no voters:
I20260619 15:01:18.696085 31859 leader_election.cc:288] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:18.696213 31864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:18.696408 31864 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [term 1 LEADER]: Becoming Leader. State: Replica: c7625d5c62444d3aa409dd2c4c0ac778, State: Running, Role: LEADER
I20260619 15:01:18.696686 31864 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37681 } }
I20260619 15:01:18.696992 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:18.684786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:51045 (local address 127.0.0.1:37681)
0619 15:01:18.685129 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:18.685148 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:18.685179 (+ 31us) server_negotiation.cc:407] Connection header received
0619 15:01:18.685768 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:18.685783 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:18.686072 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:18.686310 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:18.687183 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:18.688185 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:18.689220 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:18.689605 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:18.689750 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:18.692058 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:18.692063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:18.692072 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:18.692094 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:18.694237 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:18.694723 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:18.694731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:18.694738 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:18.694863 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:18.695147 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:18.695150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:18.695152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:18.695608 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:18.695794 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:18.696000 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 15:01:18.696221 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":125,"threads_started":1}
I20260619 15:01:18.697173 31859 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:18.697937 31865 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37681 } } }
I20260619 15:01:18.698048 31865 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:18.698406 31866 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7625d5c62444d3aa409dd2c4c0ac778. Latest consensus state: current_term: 1 leader_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7625d5c62444d3aa409dd2c4c0ac778" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37681 } } }
I20260619 15:01:18.698633 31866 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:18.699672 31870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:18.700565 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:18.702481 31870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:18.707743 31870 catalog_manager.cc:1347] Generated new cluster ID: 6cea2ca4e5e64bc3b2a66630437ecd6a
I20260619 15:01:18.707803 31870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:18.710700 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:18.702543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:52565 (local address 127.0.0.1:37681)
0619 15:01:18.702656 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:18.702661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:18.702886 (+ 225us) server_negotiation.cc:407] Connection header received
0619 15:01:18.703040 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:18.703044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:18.703088 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:18.703186 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:18.703855 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:18.704342 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:18.705057 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:18.705286 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:18.705391 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:18.706763 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:18.706766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:18.706769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:18.706792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:18.709045 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:18.709560 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:18.709563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:18.709565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:18.709640 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:18.709976 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:18.709979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:18.709981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:18.710137 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:18.710252 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:18.710496 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 15:01:18.710572 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 15:01:18.719961 31870 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:18.720738 31870 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:18.725826 31870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7625d5c62444d3aa409dd2c4c0ac778: Generated new TSK 0
I20260619 15:01:18.726398 31870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:18.729611 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31791
W20260619 15:01:18.734671 31776 connection.cc:535] client connection to 127.0.0.1:37681 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37681): Connection reset by peer (error 104)
2026-06-19T15:01:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:18 dist-test-slave-3qdh krb5kdc[31890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:18 dist-test-slave-3qdh krb5kdc[31890](info): set up 2 sockets
Jun 19 15:01:18 dist-test-slave-3qdh krb5kdc[31890](info): commencing operation
krb5kdc: starting...
W20260619 15:01:19.660130 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.912s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:19 dist-test-slave-3qdh krb5kdc[31890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:19.764775 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:19.860795 31904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:19.861109 31904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:19.861186 31904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:19.866206 31904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:19.866266 31904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:19.866289 31904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:19.866309 31904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:19.866334 31904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:19.871064 31904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:19.871728 31904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:19.872604 31904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:19.879642 31912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:19.879671 31909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:19.879766 31904 server_base.cc:1034] running on GCE node
W20260619 15:01:19.879700 31910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:19.880364 31904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:19.881206 31904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:19.882351 31904 hybrid_clock.cc:648] HybridClock initialized: now 1781881279882335 us; error 26 us; skew 500 ppm
Jun 19 15:01:19 dist-test-slave-3qdh krb5kdc[31890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:19.884843 31904 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:19.885583 31904 webserver.cc:460] Webserver started at http://127.0.0.1:39089/ using document root <none> and password file <none>
I20260619 15:01:19.886080 31904 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:19.886132 31904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:19.886358 31904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:19.888072 31904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "aaf721aa05c847e3a8f1ec812729047a"
format_stamp: "Formatted at 2026-06-19 15:01:19 on dist-test-slave-3qdh"
server_key: "e6bf1ae565ae60be31abf43f16b811bd"
server_key_iv: "9f76aa7443e1bc5a67ac627fa9e981b2"
server_key_version: "encryptionkey@0"
I20260619 15:01:19.888609 31904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "aaf721aa05c847e3a8f1ec812729047a"
format_stamp: "Formatted at 2026-06-19 15:01:19 on dist-test-slave-3qdh"
server_key: "e6bf1ae565ae60be31abf43f16b811bd"
server_key_iv: "9f76aa7443e1bc5a67ac627fa9e981b2"
server_key_version: "encryptionkey@0"
I20260619 15:01:19.891649 31904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 15:01:19.893703 31919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:19.894688 31904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:19.894809 31904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "aaf721aa05c847e3a8f1ec812729047a"
format_stamp: "Formatted at 2026-06-19 15:01:19 on dist-test-slave-3qdh"
server_key: "e6bf1ae565ae60be31abf43f16b811bd"
server_key_iv: "9f76aa7443e1bc5a67ac627fa9e981b2"
server_key_version: "encryptionkey@0"
I20260619 15:01:19.894918 31904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:19.911933 31904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:19.914368 31904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:19.914549 31904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:19.920409 31904 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38295
I20260619 15:01:19.920433 31971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38295 every 8 connection(s)
I20260619 15:01:19.921176 31904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:19.923501 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:19.928071 31972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a: Bootstrap starting.
I20260619 15:01:19.929104 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 31904
I20260619 15:01:19.929247 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:19.929509 10526 external_mini_cluster.cc:1428] Setting key cc9530cf4f844a941b81de153c923b97
I20260619 15:01:19.930450 31972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:19.931205 31972 log.cc:824] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:19.933261 31972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a: No bootstrap required, opened a new log
I20260619 15:01:19.936987 31972 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf721aa05c847e3a8f1ec812729047a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260619 15:01:19.937248 31972 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:19.937333 31972 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf721aa05c847e3a8f1ec812729047a, State: Initialized, Role: FOLLOWER
I20260619 15:01:19.937777 31972 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf721aa05c847e3a8f1ec812729047a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260619 15:01:19.937892 31972 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:19.937964 31972 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:19.938064 31972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:19.938335 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:19.930424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:40853 (local address 127.0.0.1:38295)
0619 15:01:19.930809 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:19.930824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:19.930857 (+ 33us) server_negotiation.cc:407] Connection header received
0619 15:01:19.931660 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:19.931678 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:19.932025 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:19.932367 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:19.932738 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:19.934113 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:19.934931 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:19.935622 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:19.935801 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:19.936583 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:19.936588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:19.936601 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:19.937010 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:19.937066 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:19.937109 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:19.937238 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:19.937441 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 15:01:19.937582 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260619 15:01:19.938848 31972 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf721aa05c847e3a8f1ec812729047a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260619 15:01:19.939141 31972 leader_election.cc:302] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf721aa05c847e3a8f1ec812729047a; no voters:
I20260619 15:01:19.939457 31972 leader_election.cc:288] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:19.939546 31977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:19.939716 31977 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [term 1 LEADER]: Becoming Leader. State: Replica: aaf721aa05c847e3a8f1ec812729047a, State: Running, Role: LEADER
I20260619 15:01:19.940032 31977 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf721aa05c847e3a8f1ec812729047a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260619 15:01:19.940495 31972 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:19.941192 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:19.942819 31978 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf721aa05c847e3a8f1ec812729047a. Latest consensus state: current_term: 1 leader_uuid: "aaf721aa05c847e3a8f1ec812729047a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf721aa05c847e3a8f1ec812729047a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260619 15:01:19.942955 31978 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:19.943151 31979 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaf721aa05c847e3a8f1ec812729047a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf721aa05c847e3a8f1ec812729047a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260619 15:01:19.943488 31979 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a [sys.catalog]: This master's current role is: LEADER
W20260619 15:01:19.946182 31996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:19.946259 31996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:19.946323 31997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:19.947188 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:19.943290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:43983 (local address 127.0.0.1:38295)
0619 15:01:19.943373 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:19.943377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:19.943622 (+ 245us) server_negotiation.cc:407] Connection header received
0619 15:01:19.943742 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:19.943744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:19.943774 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:19.943887 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:19.944203 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:19.944865 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:19.945581 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:19.945871 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:19.945967 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:19.946750 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:19.946754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:19.946757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:19.946768 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:19.946792 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:19.946803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:19.946885 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:19.947048 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 15:01:19.947075 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 15:01:19.947184 31997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:19.952426 31997 catalog_manager.cc:1347] Generated new cluster ID: d5781304f8ff4ab8b5417cba7f3cdcd1
I20260619 15:01:19.952486 31997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:19.959467 31997 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:19.960223 31997 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:19.970743 31997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaf721aa05c847e3a8f1ec812729047a: Generated new TSK 0
I20260619 15:01:19.971361 31997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:19.981297 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 31904
W20260619 15:01:19.986389 31888 connection.cc:535] client connection to 127.0.0.1:38295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38295): Connection reset by peer (error 104)
2026-06-19T15:01:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:20 dist-test-slave-3qdh krb5kdc[32004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:20 dist-test-slave-3qdh krb5kdc[32004](info): set up 2 sockets
Jun 19 15:01:20 dist-test-slave-3qdh krb5kdc[32004](info): commencing operation
krb5kdc: starting...
W20260619 15:01:20.920073 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:20 dist-test-slave-3qdh krb5kdc[32004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 15:01:21.025230 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:21.122939 32017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:21.123140 32017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:21.123179 32017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:21.126552 32017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:21.126591 32017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:21.126607 32017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:21.126621 32017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:21.126634 32017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:21.130983 32017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:21.131680 32017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:21.132411 32017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:21.139309 32023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:21.139309 32022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:21.139345 32025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:21.139461 32017 server_base.cc:1034] running on GCE node
I20260619 15:01:21.139986 32017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:21.140908 32017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:21.142078 32017 hybrid_clock.cc:648] HybridClock initialized: now 1781881281142058 us; error 31 us; skew 500 ppm
Jun 19 15:01:21 dist-test-slave-3qdh krb5kdc[32004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:21.144660 32017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 15:01:21.145435 32017 webserver.cc:460] Webserver started at http://127.0.0.1:37257/ using document root <none> and password file <none>
I20260619 15:01:21.145882 32017 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:21.145960 32017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:21.146147 32017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:21.147683 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "9a30937ac50a4a86985c5bc177e32664"
format_stamp: "Formatted at 2026-06-19 15:01:21 on dist-test-slave-3qdh"
server_key: "2b79d6947cd97aff14bdefe5e2c4bc44"
server_key_iv: "a2cb754028b4f16bb8f3cbef59d7afc2"
server_key_version: "encryptionkey@0"
I20260619 15:01:21.148169 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "9a30937ac50a4a86985c5bc177e32664"
format_stamp: "Formatted at 2026-06-19 15:01:21 on dist-test-slave-3qdh"
server_key: "2b79d6947cd97aff14bdefe5e2c4bc44"
server_key_iv: "a2cb754028b4f16bb8f3cbef59d7afc2"
server_key_version: "encryptionkey@0"
I20260619 15:01:21.151585 32017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 15:01:21.153733 32032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:21.154765 32017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:21.154901 32017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "9a30937ac50a4a86985c5bc177e32664"
format_stamp: "Formatted at 2026-06-19 15:01:21 on dist-test-slave-3qdh"
server_key: "2b79d6947cd97aff14bdefe5e2c4bc44"
server_key_iv: "a2cb754028b4f16bb8f3cbef59d7afc2"
server_key_version: "encryptionkey@0"
I20260619 15:01:21.155007 32017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:21.159025 32017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:21.161820 32017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:21.162029 32017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:21.168720 32017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35755
I20260619 15:01:21.168732 32084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35755 every 8 connection(s)
I20260619 15:01:21.169530 32017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:21.169710 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32017
I20260619 15:01:21.169843 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:21.170149 10526 external_mini_cluster.cc:1428] Setting key 0153fcbe56f350d53e97c5cfc8ee966e
I20260619 15:01:21.172093 32085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:21.174429 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:21.171280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:53037 (local address 127.0.0.1:35755)
0619 15:01:21.171632 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:21.171644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:21.171671 (+ 27us) server_negotiation.cc:407] Connection header received
0619 15:01:21.172141 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:21.172153 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:21.172493 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:21.172787 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:21.173466 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:21.173471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:21.173484 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:21.173502 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:21.173544 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:21.173561 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:21.173698 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:21.173762 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 15:01:21.173904 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
I20260619 15:01:21.176725 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:21.177488 32085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664: Bootstrap starting.
I20260619 15:01:21.179633 32085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:21.179792 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:21.178386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:55791 (local address 127.0.0.1:35755)
0619 15:01:21.178454 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:21.178458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:21.178653 (+ 195us) server_negotiation.cc:407] Connection header received
0619 15:01:21.178777 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:21.178780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:21.178804 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:21.178864 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:21.179448 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:21.179458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:21.179460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:21.179470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:21.179489 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:21.179501 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:21.179596 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:21.179658 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 15:01:21.179690 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 15:01:21.180399 32085 log.cc:824] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:21.181960 32085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664: No bootstrap required, opened a new log
I20260619 15:01:21.185133 32085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a30937ac50a4a86985c5bc177e32664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260619 15:01:21.185307 32085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:21.185359 32085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a30937ac50a4a86985c5bc177e32664, State: Initialized, Role: FOLLOWER
I20260619 15:01:21.185662 32085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a30937ac50a4a86985c5bc177e32664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260619 15:01:21.185758 32085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:21.185837 32085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:21.185927 32085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:21.186822 32085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a30937ac50a4a86985c5bc177e32664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260619 15:01:21.187071 32085 leader_election.cc:302] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a30937ac50a4a86985c5bc177e32664; no voters:
I20260619 15:01:21.187278 32085 leader_election.cc:288] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:21.187364 32095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:21.187618 32095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [term 1 LEADER]: Becoming Leader. State: Replica: 9a30937ac50a4a86985c5bc177e32664, State: Running, Role: LEADER
I20260619 15:01:21.187937 32095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a30937ac50a4a86985c5bc177e32664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260619 15:01:21.188290 32085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:21.189343 32096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a30937ac50a4a86985c5bc177e32664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a30937ac50a4a86985c5bc177e32664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260619 15:01:21.189469 32096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:21.189800 32097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a30937ac50a4a86985c5bc177e32664. Latest consensus state: current_term: 1 leader_uuid: "9a30937ac50a4a86985c5bc177e32664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a30937ac50a4a86985c5bc177e32664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260619 15:01:21.189916 32097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:21.190053 32104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:21.192479 32104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:21.196745 32104 catalog_manager.cc:1347] Generated new cluster ID: 97fda79b7bd64197a120545433ec73a5
I20260619 15:01:21.196805 32104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:21.219609 32104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:21.219841 32104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:21.225791 32104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a30937ac50a4a86985c5bc177e32664: Generated new TSK 0
I20260619 15:01:21.226296 32104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:21.234122 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32017
2026-06-19T15:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:21 dist-test-slave-3qdh krb5kdc[32116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:21 dist-test-slave-3qdh krb5kdc[32116](info): set up 2 sockets
Jun 19 15:01:21 dist-test-slave-3qdh krb5kdc[32116](info): commencing operation
krb5kdc: starting...
W20260619 15:01:22.189944 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:22 dist-test-slave-3qdh krb5kdc[32116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:22.304541 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:43977
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:22.396611 32129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:22.396828 32129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:22.396885 32129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:22.400260 32129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:22.400364 32129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:22.400403 32129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:22.400434 32129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:22.400471 32129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:22.404920 32129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:43977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:22.405663 32129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:22.406472 32129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:22.413288 32135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:22.413354 32134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:22.413421 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:22.413491 32129 server_base.cc:1034] running on GCE node
I20260619 15:01:22.414104 32129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:22.414991 32129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:22.416157 32129 hybrid_clock.cc:648] HybridClock initialized: now 1781881282416142 us; error 27 us; skew 500 ppm
Jun 19 15:01:22 dist-test-slave-3qdh krb5kdc[32116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:22.425063 32129 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:22.425664 32129 webserver.cc:460] Webserver started at http://127.10.71.190:45145/ using document root <none> and password file <none>
I20260619 15:01:22.426062 32129 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:22.426111 32129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:22.426237 32129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:22.427739 32129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "732f0f6af085401ebca47ec89d40157a"
format_stamp: "Formatted at 2026-06-19 15:01:22 on dist-test-slave-3qdh"
server_key: "341f50196c90e9c2583cf0098bf27626"
server_key_iv: "e2fc3a07869c68d8dd3a0834477bf284"
server_key_version: "encryptionkey@0"
I20260619 15:01:22.428154 32129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "732f0f6af085401ebca47ec89d40157a"
format_stamp: "Formatted at 2026-06-19 15:01:22 on dist-test-slave-3qdh"
server_key: "341f50196c90e9c2583cf0098bf27626"
server_key_iv: "e2fc3a07869c68d8dd3a0834477bf284"
server_key_version: "encryptionkey@0"
I20260619 15:01:22.431141 32129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 15:01:22.433195 32144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:22.434327 32129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:22.434408 32129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "732f0f6af085401ebca47ec89d40157a"
format_stamp: "Formatted at 2026-06-19 15:01:22 on dist-test-slave-3qdh"
server_key: "341f50196c90e9c2583cf0098bf27626"
server_key_iv: "e2fc3a07869c68d8dd3a0834477bf284"
server_key_version: "encryptionkey@0"
I20260619 15:01:22.434473 32129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:22.451897 32129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:22.454340 32129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:22.454458 32129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:22.460770 32129 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:43977
I20260619 15:01:22.460865 32196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:43977 every 8 connection(s)
I20260619 15:01:22.461624 32129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:22.463997 32197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:22.468743 32197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a: Bootstrap starting.
I20260619 15:01:22.468962 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32129
I20260619 15:01:22.469082 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:22.469337 10526 external_mini_cluster.cc:1428] Setting key 1e357a3346bac3e87216da23a1d85c0c
I20260619 15:01:22.470901 32197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:22.471661 32197 log.cc:824] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:22.473584 32197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a: No bootstrap required, opened a new log
I20260619 15:01:22.477047 32197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732f0f6af085401ebca47ec89d40157a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43977 } }
I20260619 15:01:22.477222 32197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:22.477301 32197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732f0f6af085401ebca47ec89d40157a, State: Initialized, Role: FOLLOWER
Jun 19 15:01:22 dist-test-slave-3qdh krb5kdc[32116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:22.477733 32197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732f0f6af085401ebca47ec89d40157a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43977 } }
I20260619 15:01:22.477886 32197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:22.477948 32197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:22.478039 32197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:22.479058 32197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732f0f6af085401ebca47ec89d40157a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43977 } }
I20260619 15:01:22.479331 32197 leader_election.cc:302] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 732f0f6af085401ebca47ec89d40157a; no voters:
I20260619 15:01:22.479597 32197 leader_election.cc:288] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:22.479653 32202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:22.479843 32202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [term 1 LEADER]: Becoming Leader. State: Replica: 732f0f6af085401ebca47ec89d40157a, State: Running, Role: LEADER
I20260619 15:01:22.480125 32202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732f0f6af085401ebca47ec89d40157a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43977 } }
I20260619 15:01:22.480659 32197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:22.481227 32204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 732f0f6af085401ebca47ec89d40157a. Latest consensus state: current_term: 1 leader_uuid: "732f0f6af085401ebca47ec89d40157a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732f0f6af085401ebca47ec89d40157a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43977 } } }
I20260619 15:01:22.481397 32204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:22.481676 32203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "732f0f6af085401ebca47ec89d40157a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732f0f6af085401ebca47ec89d40157a" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43977 } } }
I20260619 15:01:22.481776 32203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:22.481832 32210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:22.483438 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:22.470439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32885 (local address 127.10.71.190:43977)
0619 15:01:22.470799 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:22.470816 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:22.470852 (+ 36us) server_negotiation.cc:407] Connection header received
0619 15:01:22.471619 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:22.471638 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:22.472000 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:22.472243 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:22.472981 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:22.474357 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:22.475104 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:22.475730 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:22.475911 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:22.478329 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:22.478334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:22.478347 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:22.478373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:22.480117 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:22.480712 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:22.480718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:22.480723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:22.480882 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:22.481250 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:22.481253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:22.481255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:22.481719 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:22.481907 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:22.482400 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 15:01:22.482610 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260619 15:01:22.485282 32210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:22.486864 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:22.490236 32210 catalog_manager.cc:1347] Generated new cluster ID: 889a774e0883418f9678dce4f75d5145
I20260619 15:01:22.490286 32210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:22.496804 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:22.488869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59635 (local address 127.10.71.190:43977)
0619 15:01:22.488954 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:22.488958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:22.489217 (+ 259us) server_negotiation.cc:407] Connection header received
0619 15:01:22.489309 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:22.489314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:22.489350 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:22.489422 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:22.489979 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:22.490581 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:22.491604 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:22.491905 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:22.491989 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:22.493470 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:22.493473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:22.493476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:22.493496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:22.495022 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:22.495582 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:22.495585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:22.495586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:22.495656 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:22.495991 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:22.495994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:22.495995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:22.496152 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:22.496279 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:22.496594 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 15:01:22.496666 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 15:01:22.506956 32210 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:22.507843 32210 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:22.514266 32210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 732f0f6af085401ebca47ec89d40157a: Generated new TSK 0
I20260619 15:01:22.514778 32210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:22.523530 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32129
W20260619 15:01:22.528501 32114 connection.cc:535] client connection to 127.10.71.190:43977 recv error: Network error: failed to read from TLS socket (remote: 127.10.71.190:43977): Connection reset by peer (error 104)
2026-06-19T15:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:22 dist-test-slave-3qdh krb5kdc[32228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:22 dist-test-slave-3qdh krb5kdc[32228](info): set up 2 sockets
Jun 19 15:01:22 dist-test-slave-3qdh krb5kdc[32228](info): commencing operation
krb5kdc: starting...
W20260619 15:01:23.452277 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.909s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:23 dist-test-slave-3qdh krb5kdc[32228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:23.555371 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40013
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:23.649658 32241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:23.649840 32241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:23.649873 32241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:23.653402 32241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:23.653452 32241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:23.653472 32241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:23.653528 32241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:23.653564 32241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:23.657860 32241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:40013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:23.658581 32241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:23.659373 32241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:23.666330 32247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:23.666421 32241 server_base.cc:1034] running on GCE node
W20260619 15:01:23.666329 32249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:23.666311 32246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:23.667062 32241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:23.667881 32241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:23.669056 32241 hybrid_clock.cc:648] HybridClock initialized: now 1781881283669053 us; error 38 us; skew 500 ppm
Jun 19 15:01:23 dist-test-slave-3qdh krb5kdc[32228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:23.671301 32241 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:23.672003 32241 webserver.cc:460] Webserver started at http://127.10.71.190:42985/ using document root <none> and password file <none>
I20260619 15:01:23.672488 32241 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:23.672561 32241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:23.672744 32241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:23.674430 32241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "98e8b2c73824495184d399f707f6db93"
format_stamp: "Formatted at 2026-06-19 15:01:23 on dist-test-slave-3qdh"
server_key: "1a2bf15219029407fc6444c85011234a"
server_key_iv: "43e91198563be9be20aad3e785151f72"
server_key_version: "encryptionkey@0"
I20260619 15:01:23.674881 32241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "98e8b2c73824495184d399f707f6db93"
format_stamp: "Formatted at 2026-06-19 15:01:23 on dist-test-slave-3qdh"
server_key: "1a2bf15219029407fc6444c85011234a"
server_key_iv: "43e91198563be9be20aad3e785151f72"
server_key_version: "encryptionkey@0"
I20260619 15:01:23.677754 32241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 15:01:23.679607 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:23.680600 32241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:23.680711 32241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "98e8b2c73824495184d399f707f6db93"
format_stamp: "Formatted at 2026-06-19 15:01:23 on dist-test-slave-3qdh"
server_key: "1a2bf15219029407fc6444c85011234a"
server_key_iv: "43e91198563be9be20aad3e785151f72"
server_key_version: "encryptionkey@0"
I20260619 15:01:23.680799 32241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:23.697820 32241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:23.700001 32241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:23.700169 32241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:23.705826 32241 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:40013
I20260619 15:01:23.705840 32308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:40013 every 8 connection(s)
I20260619 15:01:23.706581 32241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:23.708837 32309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:23.709576 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32241
I20260619 15:01:23.709712 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:23.709935 10526 external_mini_cluster.cc:1428] Setting key 3001db783328be2dd64e6ee27a3b0960
I20260619 15:01:23.714332 32309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93: Bootstrap starting.
I20260619 15:01:23.716470 32309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:23.717267 32309 log.cc:824] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:23.718366 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:23.711001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40229 (local address 127.10.71.190:40013)
0619 15:01:23.711352 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:23.711371 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:23.711407 (+ 36us) server_negotiation.cc:407] Connection header received
0619 15:01:23.712085 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:23.712104 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:23.712455 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:23.712768 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:23.713134 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:23.714487 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:23.715238 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:23.715801 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:23.715936 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:23.716676 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:23.716682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:23.716697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:23.717116 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:23.717182 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:23.717208 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:23.717329 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:23.717606 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 15:01:23.717714 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260619 15:01:23.718981 32309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93: No bootstrap required, opened a new log
I20260619 15:01:23.720940 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:23.722702 32309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8b2c73824495184d399f707f6db93" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40013 } }
I20260619 15:01:23.722931 32309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:23.723022 32309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e8b2c73824495184d399f707f6db93, State: Initialized, Role: FOLLOWER
I20260619 15:01:23.723467 32309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8b2c73824495184d399f707f6db93" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40013 } }
I20260619 15:01:23.723619 32309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:23.723683 32309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:23.723770 32309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:23.724689 32309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8b2c73824495184d399f707f6db93" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40013 } }
I20260619 15:01:23.724973 32309 leader_election.cc:302] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98e8b2c73824495184d399f707f6db93; no voters:
I20260619 15:01:23.725239 32309 leader_election.cc:288] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:23.725340 32319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:23.725575 32319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [term 1 LEADER]: Becoming Leader. State: Replica: 98e8b2c73824495184d399f707f6db93, State: Running, Role: LEADER
I20260619 15:01:23.725978 32319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8b2c73824495184d399f707f6db93" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40013 } }
I20260619 15:01:23.726414 32309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:23.726648 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:23.722625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36745 (local address 127.10.71.190:40013)
0619 15:01:23.722724 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:23.722728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:23.722876 (+ 148us) server_negotiation.cc:407] Connection header received
0619 15:01:23.723017 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:23.723025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:23.723075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:23.723152 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:23.723427 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:23.724173 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:23.724965 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:23.725301 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:23.725403 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:23.726181 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:23.726184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:23.726187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:23.726198 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:23.726220 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:23.726231 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:23.726314 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:23.726509 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 15:01:23.726539 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 15:01:23.727079 32319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e8b2c73824495184d399f707f6db93. Latest consensus state: current_term: 1 leader_uuid: "98e8b2c73824495184d399f707f6db93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8b2c73824495184d399f707f6db93" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40013 } } }
I20260619 15:01:23.727716 32319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:23.727597 32321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98e8b2c73824495184d399f707f6db93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e8b2c73824495184d399f707f6db93" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40013 } } }
I20260619 15:01:23.728008 32321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:23.728132 32328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:23.730340 32328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:23.734869 32328 catalog_manager.cc:1347] Generated new cluster ID: 543b9c4db0494d5894d18d13f2d4974a
I20260619 15:01:23.734922 32328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:23.746699 32328 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:23.747385 32328 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:23.751365 32328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e8b2c73824495184d399f707f6db93: Generated new TSK 0
I20260619 15:01:23.751837 32328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:23.755149 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32241
W20260619 15:01:23.759394 32226 connection.cc:535] client connection to 127.10.71.190:40013 recv error: Network error: failed to read from TLS socket (remote: 127.10.71.190:40013): Connection reset by peer (error 104)
2026-06-19T15:01:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:23 dist-test-slave-3qdh krb5kdc[32340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:23 dist-test-slave-3qdh krb5kdc[32340](info): set up 2 sockets
Jun 19 15:01:23 dist-test-slave-3qdh krb5kdc[32340](info): commencing operation
krb5kdc: starting...
W20260619 15:01:24.716367 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:24 dist-test-slave-3qdh krb5kdc[32340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:24.819207 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37381
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:24.913786 32353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:24.914018 32353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:24.914094 32353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:24.917713 32353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:24.917784 32353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:24.917819 32353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:24.917868 32353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:24.917904 32353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:24.922408 32353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:37381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:24.923229 32353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:24.924052 32353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:01:24.931318 32353 server_base.cc:1034] running on GCE node
W20260619 15:01:24.931267 32358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:24.931275 32359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:24.931275 32361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:24.931978 32353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:24.932889 32353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:24.934108 32353 hybrid_clock.cc:648] HybridClock initialized: now 1781881284934098 us; error 34 us; skew 500 ppm
Jun 19 15:01:24 dist-test-slave-3qdh krb5kdc[32340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:24.936399 32353 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:24.937047 32353 webserver.cc:460] Webserver started at http://127.10.71.190:38863/ using document root <none> and password file <none>
I20260619 15:01:24.937448 32353 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:24.937495 32353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:24.937623 32353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:24.939200 32353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "f20ab06da7524d6bad0a1b4908b98bd4"
format_stamp: "Formatted at 2026-06-19 15:01:24 on dist-test-slave-3qdh"
server_key: "fe356922f2ede2f5a33e03af6f37a08d"
server_key_iv: "6cb2816b79fd9d2bc17991ac1008ae8b"
server_key_version: "encryptionkey@0"
I20260619 15:01:24.939623 32353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "f20ab06da7524d6bad0a1b4908b98bd4"
format_stamp: "Formatted at 2026-06-19 15:01:24 on dist-test-slave-3qdh"
server_key: "fe356922f2ede2f5a33e03af6f37a08d"
server_key_iv: "6cb2816b79fd9d2bc17991ac1008ae8b"
server_key_version: "encryptionkey@0"
I20260619 15:01:24.942711 32353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 15:01:24.944847 32368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:24.945855 32353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 15:01:24.945960 32353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "f20ab06da7524d6bad0a1b4908b98bd4"
format_stamp: "Formatted at 2026-06-19 15:01:24 on dist-test-slave-3qdh"
server_key: "fe356922f2ede2f5a33e03af6f37a08d"
server_key_iv: "6cb2816b79fd9d2bc17991ac1008ae8b"
server_key_version: "encryptionkey@0"
I20260619 15:01:24.946054 32353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:24.949910 32353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:24.952189 32353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:24.952374 32353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:24.958468 32353 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:37381
I20260619 15:01:24.958479 32421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:37381 every 8 connection(s)
I20260619 15:01:24.959203 32353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:24.961923 32422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:24.963533 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32353
I20260619 15:01:24.963677 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:24.963959 10526 external_mini_cluster.cc:1428] Setting key d41f4308d8c7c8df89142985451d8aa7
I20260619 15:01:24.966894 32422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4: Bootstrap starting.
I20260619 15:01:24.968528 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:24.965120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45013 (local address 127.10.71.190:37381)
0619 15:01:24.965480 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:24.965493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:24.965521 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:01:24.966226 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:24.966241 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:24.966501 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:24.966786 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:24.967403 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:24.967408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:24.967421 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:24.967436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:24.967484 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:24.967500 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:24.967632 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:24.967707 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 15:01:24.967835 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260619 15:01:24.969179 32422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:24.969918 32422 log.cc:824] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:24.971160 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:24.971870 32422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4: No bootstrap required, opened a new log
I20260619 15:01:24.974792 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:24.973271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52677 (local address 127.10.71.190:37381)
0619 15:01:24.973375 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:24.973379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:24.973564 (+ 185us) server_negotiation.cc:407] Connection header received
0619 15:01:24.973654 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:24.973657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:24.973686 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:24.973763 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:24.974400 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:24.974403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:24.974406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:24.974416 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:24.974439 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:24.974450 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:24.974541 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:24.974649 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 15:01:24.974679 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 15:01:24.975977 32422 raft_consensus.cc:348] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37381 } }
I20260619 15:01:24.976153 32422 raft_consensus.cc:374] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:24.976184 32422 raft_consensus.cc:729] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20ab06da7524d6bad0a1b4908b98bd4, State: Initialized, Role: FOLLOWER
I20260619 15:01:24.976557 32422 consensus_queue.cc:260] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37381 } }
I20260619 15:01:24.976658 32422 raft_consensus.cc:388] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:24.976697 32422 raft_consensus.cc:482] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:24.976758 32422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:24.977427 32422 raft_consensus.cc:504] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37381 } }
I20260619 15:01:24.977670 32422 leader_election.cc:302] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f20ab06da7524d6bad0a1b4908b98bd4; no voters:
I20260619 15:01:24.977880 32422 leader_election.cc:288] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:24.977974 32432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:24.978173 32432 raft_consensus.cc:686] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [term 1 LEADER]: Becoming Leader. State: Replica: f20ab06da7524d6bad0a1b4908b98bd4, State: Running, Role: LEADER
I20260619 15:01:24.978471 32432 consensus_queue.cc:237] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37381 } }
I20260619 15:01:24.978892 32422 sys_catalog.cc:564] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:24.979895 32434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f20ab06da7524d6bad0a1b4908b98bd4. Latest consensus state: current_term: 1 leader_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37381 } } }
I20260619 15:01:24.980006 32434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:24.980594 32433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20ab06da7524d6bad0a1b4908b98bd4" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 37381 } } }
I20260619 15:01:24.980693 32433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:24.981038 32441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:24.983283 32441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:24.987901 32441 catalog_manager.cc:1347] Generated new cluster ID: 71d1b3b410f144aba3e33f0582f70e2e
I20260619 15:01:24.987957 32441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:25.002449 32441 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:25.002705 32441 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:25.017504 32441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f20ab06da7524d6bad0a1b4908b98bd4: Generated new TSK 0
I20260619 15:01:25.018087 32441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:25.030342 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32353
2026-06-19T15:01:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:25 dist-test-slave-3qdh krb5kdc[32453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:25 dist-test-slave-3qdh krb5kdc[32453](info): set up 2 sockets
Jun 19 15:01:25 dist-test-slave-3qdh krb5kdc[32453](info): commencing operation
krb5kdc: starting...
W20260619 15:01:25.964015 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:26 dist-test-slave-3qdh krb5kdc[32453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:26.068763 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39057
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:26.163743 32466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:26.163944 32466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:26.163982 32466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:26.167413 32466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:26.167456 32466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:26.167474 32466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:26.167486 32466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:26.167500 32466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:26.172356 32466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:39057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:26.173121 32466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:26.173930 32466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:26.181128 32472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:26.181142 32474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:26.181219 32466 server_base.cc:1034] running on GCE node
W20260619 15:01:26.181131 32471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:26.181756 32466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:26.182663 32466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:26.183851 32466 hybrid_clock.cc:648] HybridClock initialized: now 1781881286183836 us; error 26 us; skew 500 ppm
Jun 19 15:01:26 dist-test-slave-3qdh krb5kdc[32453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881286, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:26.186398 32466 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:26.187062 32466 webserver.cc:460] Webserver started at http://127.10.71.190:36185/ using document root <none> and password file <none>
I20260619 15:01:26.187476 32466 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:26.187526 32466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:26.187657 32466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:26.189414 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "bb3fb02b01144dc683d202c995890d7b"
format_stamp: "Formatted at 2026-06-19 15:01:26 on dist-test-slave-3qdh"
server_key: "1663adbf10f1c89e79d5d115e8c21e06"
server_key_iv: "f479d8ee6f8a63a304ded1aadaae909c"
server_key_version: "encryptionkey@0"
I20260619 15:01:26.189867 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "bb3fb02b01144dc683d202c995890d7b"
format_stamp: "Formatted at 2026-06-19 15:01:26 on dist-test-slave-3qdh"
server_key: "1663adbf10f1c89e79d5d115e8c21e06"
server_key_iv: "f479d8ee6f8a63a304ded1aadaae909c"
server_key_version: "encryptionkey@0"
I20260619 15:01:26.192832 32466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:01:26.194882 32481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:26.195870 32466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:26.195969 32466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "bb3fb02b01144dc683d202c995890d7b"
format_stamp: "Formatted at 2026-06-19 15:01:26 on dist-test-slave-3qdh"
server_key: "1663adbf10f1c89e79d5d115e8c21e06"
server_key_iv: "f479d8ee6f8a63a304ded1aadaae909c"
server_key_version: "encryptionkey@0"
I20260619 15:01:26.196039 32466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:26.206759 32466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:26.209399 32466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:26.209517 32466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:26.215526 32466 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:39057
I20260619 15:01:26.215545 32533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:39057 every 8 connection(s)
I20260619 15:01:26.216274 32466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:26.218602 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:26.222968 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32466
I20260619 15:01:26.223095 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:26.223387 10526 external_mini_cluster.cc:1428] Setting key 3c4987953adbe2b453fffb3fc2e8342c
I20260619 15:01:26.223503 32534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b: Bootstrap starting.
I20260619 15:01:26.225662 32534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:26.226558 32534 log.cc:824] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:26.228331 32534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b: No bootstrap required, opened a new log
Jun 19 15:01:26 dist-test-slave-3qdh krb5kdc[32453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:26.232084 32534 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3fb02b01144dc683d202c995890d7b" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39057 } }
I20260619 15:01:26.232354 32534 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:26.232452 32534 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3fb02b01144dc683d202c995890d7b, State: Initialized, Role: FOLLOWER
I20260619 15:01:26.232892 32534 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3fb02b01144dc683d202c995890d7b" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39057 } }
I20260619 15:01:26.232995 32534 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:26.233117 32534 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:26.233244 32534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:26.233992 32534 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3fb02b01144dc683d202c995890d7b" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39057 } }
I20260619 15:01:26.234303 32534 leader_election.cc:302] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb3fb02b01144dc683d202c995890d7b; no voters:
I20260619 15:01:26.234601 32534 leader_election.cc:288] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:26.234654 32539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:26.234862 32539 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [term 1 LEADER]: Becoming Leader. State: Replica: bb3fb02b01144dc683d202c995890d7b, State: Running, Role: LEADER
I20260619 15:01:26.235103 32539 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3fb02b01144dc683d202c995890d7b" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39057 } }
I20260619 15:01:26.235755 32534 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:26.236182 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb3fb02b01144dc683d202c995890d7b. Latest consensus state: current_term: 1 leader_uuid: "bb3fb02b01144dc683d202c995890d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3fb02b01144dc683d202c995890d7b" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39057 } } }
I20260619 15:01:26.236287 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:26.236513 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb3fb02b01144dc683d202c995890d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3fb02b01144dc683d202c995890d7b" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 39057 } } }
I20260619 15:01:26.236682 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:26.236720 32543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:26.237146 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:26.224555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44017 (local address 127.10.71.190:39057)
0619 15:01:26.224882 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:26.224895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:26.224925 (+ 30us) server_negotiation.cc:407] Connection header received
0619 15:01:26.225460 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:26.225474 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:26.225786 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:26.226051 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:26.227008 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:26.228001 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:26.229094 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:26.229494 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:26.229600 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:26.232116 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:26.232120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:26.232131 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:26.232154 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:26.233903 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:26.234414 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:26.234419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:26.234426 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:26.234552 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:26.234891 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:26.234895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:26.234897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:26.235356 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:26.235560 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:26.235850 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 15:01:26.236232 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260619 15:01:26.239632 32543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:26.239974 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:26.244859 32543 catalog_manager.cc:1347] Generated new cluster ID: 21b0444e34f044b4be5acb558c3b8426
I20260619 15:01:26.244976 32543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:26.249502 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:26.241869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60117 (local address 127.10.71.190:39057)
0619 15:01:26.241970 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:26.241974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:26.242172 (+ 198us) server_negotiation.cc:407] Connection header received
0619 15:01:26.242253 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:26.242255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:26.242293 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:26.242381 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:26.243018 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:26.243765 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:26.244783 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:26.245132 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:26.245229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:26.246534 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:26.246538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:26.246540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:26.246562 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:26.247956 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:26.248404 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:26.248407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:26.248409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:26.248491 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:26.248759 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:26.248763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:26.248764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:26.248961 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:26.249073 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:26.249284 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 15:01:26.249378 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 15:01:26.258607 32543 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:26.259317 32543 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:26.271540 32543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb3fb02b01144dc683d202c995890d7b: Generated new TSK 0
I20260619 15:01:26.272063 32543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:26.284830 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32466
2026-06-19T15:01:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:26 dist-test-slave-3qdh krb5kdc[32565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:26 dist-test-slave-3qdh krb5kdc[32565](info): set up 2 sockets
Jun 19 15:01:26 dist-test-slave-3qdh krb5kdc[32565](info): commencing operation
krb5kdc: starting...
W20260619 15:01:27.219998 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:27 dist-test-slave-3qdh krb5kdc[32565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:27.320950 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40397
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:38337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:27.415611 32578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:27.415819 32578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:27.415884 32578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:27.419203 32578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:27.419271 32578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:27.419303 32578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:27.419346 32578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:27.419392 32578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:27.423859 32578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:38337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:27.424743 32578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:27.425535 32578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:27.432596 32586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:27.432674 32583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:27.432699 32578 server_base.cc:1034] running on GCE node
W20260619 15:01:27.432610 32584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:27.433323 32578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:27.434265 32578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:27.435442 32578 hybrid_clock.cc:648] HybridClock initialized: now 1781881287435421 us; error 115 us; skew 500 ppm
Jun 19 15:01:27 dist-test-slave-3qdh krb5kdc[32565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:27.440186 32578 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:27.440975 32578 webserver.cc:460] Webserver started at http://127.10.71.190:36147/ using document root <none> and password file <none>
I20260619 15:01:27.441408 32578 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:27.441499 32578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:27.441705 32578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:27.443312 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "6a93517308444d9b94be8c4b128b2e92"
format_stamp: "Formatted at 2026-06-19 15:01:27 on dist-test-slave-3qdh"
server_key: "ac0de76aa4744f202a73a2f46d492b59"
server_key_iv: "bb22eb236d0f0e5809e7f813051790ac"
server_key_version: "encryptionkey@0"
I20260619 15:01:27.443823 32578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "6a93517308444d9b94be8c4b128b2e92"
format_stamp: "Formatted at 2026-06-19 15:01:27 on dist-test-slave-3qdh"
server_key: "ac0de76aa4744f202a73a2f46d492b59"
server_key_iv: "bb22eb236d0f0e5809e7f813051790ac"
server_key_version: "encryptionkey@0"
I20260619 15:01:27.446852 32578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:27.448865 32593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:27.450176 32578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:27.450291 32578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "6a93517308444d9b94be8c4b128b2e92"
format_stamp: "Formatted at 2026-06-19 15:01:27 on dist-test-slave-3qdh"
server_key: "ac0de76aa4744f202a73a2f46d492b59"
server_key_iv: "bb22eb236d0f0e5809e7f813051790ac"
server_key_version: "encryptionkey@0"
I20260619 15:01:27.450398 32578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:27.463688 32578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:27.466518 32578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:27.466694 32578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:27.472435 32578 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:40397
I20260619 15:01:27.472458 32645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:40397 every 8 connection(s)
I20260619 15:01:27.473181 32578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:27.475144 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32578
I20260619 15:01:27.475260 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:27.475504 10526 external_mini_cluster.cc:1428] Setting key 8627cd408e5e650a005988de47630173
I20260619 15:01:27.475551 32646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:27.480751 32646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92: Bootstrap starting.
I20260619 15:01:27.482420 32646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:27.483065 32646 log.cc:824] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:27.483094 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:27.476762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56433 (local address 127.10.71.190:40397)
0619 15:01:27.477132 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:27.477148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:27.477176 (+ 28us) server_negotiation.cc:407] Connection header received
0619 15:01:27.477893 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:27.477908 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:27.478212 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:27.478514 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:27.478858 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:27.479657 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:27.480506 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:27.480944 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:27.481085 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:27.481796 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:27.481806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:27.481816 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:27.482109 (+ 293us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:27.482151 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:27.482170 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:27.482267 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:27.482365 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 15:01:27.482544 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":124,"threads_started":1}
I20260619 15:01:27.485024 32646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92: No bootstrap required, opened a new log
I20260619 15:01:27.485625 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:27.488593 32646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a93517308444d9b94be8c4b128b2e92" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40397 } }
I20260619 15:01:27.488811 32646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:27.488879 32646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a93517308444d9b94be8c4b128b2e92, State: Initialized, Role: FOLLOWER
I20260619 15:01:27.489249 32646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a93517308444d9b94be8c4b128b2e92" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40397 } }
I20260619 15:01:27.489378 32646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:27.489434 32646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:27.489508 32646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:27.490298 32646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a93517308444d9b94be8c4b128b2e92" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40397 } }
I20260619 15:01:27.490621 32646 leader_election.cc:302] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a93517308444d9b94be8c4b128b2e92; no voters:
I20260619 15:01:27.490751 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:27.487271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54653 (local address 127.10.71.190:40397)
0619 15:01:27.487378 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:27.487382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:27.487484 (+ 102us) server_negotiation.cc:407] Connection header received
0619 15:01:27.487552 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:27.487555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:27.487591 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:27.487685 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:27.487868 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:27.488584 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:27.489350 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:27.489673 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:27.489778 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:27.490416 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:27.490419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:27.490421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:27.490432 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:27.490454 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:27.490465 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:27.490535 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:27.490595 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 15:01:27.490639 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 15:01:27.490969 32646 leader_election.cc:288] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:27.491035 32656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:27.491214 32656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [term 1 LEADER]: Becoming Leader. State: Replica: 6a93517308444d9b94be8c4b128b2e92, State: Running, Role: LEADER
I20260619 15:01:27.491526 32656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a93517308444d9b94be8c4b128b2e92" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40397 } }
I20260619 15:01:27.492056 32646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:27.492662 32658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a93517308444d9b94be8c4b128b2e92. Latest consensus state: current_term: 1 leader_uuid: "6a93517308444d9b94be8c4b128b2e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a93517308444d9b94be8c4b128b2e92" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40397 } } }
I20260619 15:01:27.492892 32658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:27.493041 32657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a93517308444d9b94be8c4b128b2e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a93517308444d9b94be8c4b128b2e92" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 40397 } } }
I20260619 15:01:27.493145 32657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:27.493328 32665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:27.496110 32665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:27.500568 32665 catalog_manager.cc:1347] Generated new cluster ID: dcb4d2607bcd4d609f265d06046f430d
I20260619 15:01:27.500631 32665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:27.512780 32665 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:27.513803 32665 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:27.518342 32665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a93517308444d9b94be8c4b128b2e92: Generated new TSK 0
I20260619 15:01:27.518949 32665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:27.522867 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32578
2026-06-19T15:01:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:27 dist-test-slave-3qdh krb5kdc[32677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:27 dist-test-slave-3qdh krb5kdc[32677](info): set up 2 sockets
Jun 19 15:01:27 dist-test-slave-3qdh krb5kdc[32677](info): commencing operation
krb5kdc: starting...
W20260619 15:01:28.468214 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:28 dist-test-slave-3qdh krb5kdc[32677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:28.569128 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38777
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:28.664580 32690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:28.664759 32690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:28.664793 32690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:28.668188 32690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:28.668231 32690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:28.668247 32690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:28.668260 32690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:28.668298 32690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:28.672526 32690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:28.673291 32690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:28.674144 32690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:28.680830 32698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:28.680804 32696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:28.680815 32695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:28.680989 32690 server_base.cc:1034] running on GCE node
I20260619 15:01:28.681502 32690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:28.682320 32690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:28.683494 32690 hybrid_clock.cc:648] HybridClock initialized: now 1781881288683479 us; error 33 us; skew 500 ppm
Jun 19 15:01:28 dist-test-slave-3qdh krb5kdc[32677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:28.685803 32690 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:28.686502 32690 webserver.cc:460] Webserver started at http://127.10.71.190:36033/ using document root <none> and password file <none>
I20260619 15:01:28.686909 32690 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:28.686957 32690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:28.687088 32690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:28.688707 32690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "9ee8e118609840948d242dc3fefca6fd"
format_stamp: "Formatted at 2026-06-19 15:01:28 on dist-test-slave-3qdh"
server_key: "de4fb9e9f21e0a2ef95f3771df5d6135"
server_key_iv: "ad7174ca4b01d7534285472a8090d213"
server_key_version: "encryptionkey@0"
I20260619 15:01:28.689118 32690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "9ee8e118609840948d242dc3fefca6fd"
format_stamp: "Formatted at 2026-06-19 15:01:28 on dist-test-slave-3qdh"
server_key: "de4fb9e9f21e0a2ef95f3771df5d6135"
server_key_iv: "ad7174ca4b01d7534285472a8090d213"
server_key_version: "encryptionkey@0"
I20260619 15:01:28.692015 32690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 15:01:28.694002 32705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:28.695029 32690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 15:01:28.695135 32690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "9ee8e118609840948d242dc3fefca6fd"
format_stamp: "Formatted at 2026-06-19 15:01:28 on dist-test-slave-3qdh"
server_key: "de4fb9e9f21e0a2ef95f3771df5d6135"
server_key_iv: "ad7174ca4b01d7534285472a8090d213"
server_key_version: "encryptionkey@0"
I20260619 15:01:28.695216 32690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:28.699040 32690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:28.701383 32690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:28.701542 32690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:28.707376 32690 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:38777
I20260619 15:01:28.707386 32757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:38777 every 8 connection(s)
I20260619 15:01:28.708144 32690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:28.710415 32758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:28.713707 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 32690
I20260619 15:01:28.713845 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:28.714115 10526 external_mini_cluster.cc:1428] Setting key f46593c3d8342004d3751d5bf5774b1f
I20260619 15:01:28.715550 32758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd: Bootstrap starting.
I20260619 15:01:28.717664 32758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:28.718372 32758 log.cc:824] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:28.718472 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:28.715273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33019 (local address 127.10.71.190:38777)
0619 15:01:28.715618 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:28.715636 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:28.715677 (+ 41us) server_negotiation.cc:407] Connection header received
0619 15:01:28.716179 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:28.716192 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:28.716477 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:28.716714 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:28.717439 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:28.717443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:28.717452 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:28.717460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:28.717502 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:28.717518 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:28.717596 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:28.717752 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 15:01:28.717874 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
I20260619 15:01:28.720206 32758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd: No bootstrap required, opened a new log
I20260619 15:01:28.721032 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:28.724066 32758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8e118609840948d242dc3fefca6fd" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38777 } }
I20260619 15:01:28.724366 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:28.722864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40115 (local address 127.10.71.190:38777)
0619 15:01:28.722961 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:28.722964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:28.723101 (+ 137us) server_negotiation.cc:407] Connection header received
0619 15:01:28.723204 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:28.723206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:28.723234 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:28.723329 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:28.723964 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:28.723966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:28.723968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 15:01:28.723977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:28.723998 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 15:01:28.724007 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:28.724073 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:28.724183 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 15:01:28.724211 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 15:01:28.724350 32758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 15:01:28.724473 32758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee8e118609840948d242dc3fefca6fd, State: Initialized, Role: FOLLOWER
I20260619 15:01:28.724929 32758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8e118609840948d242dc3fefca6fd" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38777 } }
I20260619 15:01:28.725097 32758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 15:01:28.725176 32758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 15:01:28.725278 32758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:28.726105 32758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8e118609840948d242dc3fefca6fd" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38777 } }
I20260619 15:01:28.726435 32758 leader_election.cc:302] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ee8e118609840948d242dc3fefca6fd; no voters:
I20260619 15:01:28.726707 32758 leader_election.cc:288] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 15:01:28.726837 300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:28.727046 300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [term 1 LEADER]: Becoming Leader. State: Replica: 9ee8e118609840948d242dc3fefca6fd, State: Running, Role: LEADER
I20260619 15:01:28.727342 300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8e118609840948d242dc3fefca6fd" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38777 } }
I20260619 15:01:28.727880 32758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:28.728541 301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ee8e118609840948d242dc3fefca6fd. Latest consensus state: current_term: 1 leader_uuid: "9ee8e118609840948d242dc3fefca6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8e118609840948d242dc3fefca6fd" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38777 } } }
I20260619 15:01:28.728734 301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:28.728888 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ee8e118609840948d242dc3fefca6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8e118609840948d242dc3fefca6fd" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 38777 } } }
I20260619 15:01:28.728984 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:28.729180 309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:28.731823 309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:28.736918 309 catalog_manager.cc:1347] Generated new cluster ID: ca3c7d92f25949ffa92e7c2ca5a2bd05
I20260619 15:01:28.736979 309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:28.771365 309 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:28.771701 309 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:28.783463 309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ee8e118609840948d242dc3fefca6fd: Generated new TSK 0
I20260619 15:01:28.784163 309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 15:01:28.790508 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 32690
2026-06-19T15:01:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1268 ms)
[----------] 18 tests from AuthTokenIssuingTest (22863 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:28 dist-test-slave-3qdh krb5kdc[321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:28 dist-test-slave-3qdh krb5kdc[321](info): set up 2 sockets
Jun 19 15:01:28 dist-test-slave-3qdh krb5kdc[321](info): commencing operation
krb5kdc: starting...
W20260619 15:01:29.744166 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:29 dist-test-slave-3qdh krb5kdc[321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:29.846491 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:45827
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:45827,127.10.71.189:37857,127.10.71.188:34785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:29.941823 334 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:01:29.942088 334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:29.942154 334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:29.942224 334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:29.945938 334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:29.946020 334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:29.946058 334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:29.946091 334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:29.946120 334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:29.950713 334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:45827,127.10.71.189:37857,127.10.71.188:34785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:45827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:29.951591 334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:29.952504 334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:01:29.959988 334 server_base.cc:1034] running on GCE node
W20260619 15:01:29.959959 340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:29.959956 339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:29.960062 342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:29.960767 334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:29.961607 334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:29.962759 334 hybrid_clock.cc:648] HybridClock initialized: now 1781881289962742 us; error 28 us; skew 500 ppm
Jun 19 15:01:29 dist-test-slave-3qdh krb5kdc[321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:29.965221 334 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:29.965879 334 webserver.cc:460] Webserver started at http://127.10.71.190:41295/ using document root <none> and password file <none>
I20260619 15:01:29.966284 334 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:29.966332 334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:29.966461 334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:29.968043 334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "55535589ac8b42e6a8d5093343a1a353"
format_stamp: "Formatted at 2026-06-19 15:01:29 on dist-test-slave-3qdh"
server_key: "fef7ff72e5d7a85683153cb4fa510be3"
server_key_iv: "8d7e9b6827e1c29d0259d4f50c474728"
server_key_version: "encryptionkey@0"
I20260619 15:01:29.968605 334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "55535589ac8b42e6a8d5093343a1a353"
format_stamp: "Formatted at 2026-06-19 15:01:29 on dist-test-slave-3qdh"
server_key: "fef7ff72e5d7a85683153cb4fa510be3"
server_key_iv: "8d7e9b6827e1c29d0259d4f50c474728"
server_key_version: "encryptionkey@0"
I20260619 15:01:29.971701 334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 15:01:29.973960 349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:29.975073 334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 15:01:29.975185 334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "55535589ac8b42e6a8d5093343a1a353"
format_stamp: "Formatted at 2026-06-19 15:01:29 on dist-test-slave-3qdh"
server_key: "fef7ff72e5d7a85683153cb4fa510be3"
server_key_iv: "8d7e9b6827e1c29d0259d4f50c474728"
server_key_version: "encryptionkey@0"
I20260619 15:01:29.975272 334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:29.989564 334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:29.992100 334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:29.992280 334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:29.998343 334 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:45827
I20260619 15:01:29.998582 401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:45827 every 8 connection(s)
I20260619 15:01:29.999126 334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:30.001179 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 334
I20260619 15:01:30.001293 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:30.001559 10526 external_mini_cluster.cc:1428] Setting key d4ddd558cffd827ca93f169ed07b21c9
I20260619 15:01:30.001667 402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:30.005398 402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.71.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.189@KRBTEST.COM" created.
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:30.019167 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.007235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45827 (local address 127.0.0.1:41238)
0619 15:01:30.007718 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.007753 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.008514 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.009131 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.009135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.009478 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.009683 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.009694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.010900 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.010907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.011797 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.011810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.012440 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.012446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.012601 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.013134 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.013148 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.014931 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.016738 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.016749 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.016757 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.017036 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.017335 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.017339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.017343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.017479 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.017963 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.017971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.018112 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.018277 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.018588 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":107,"spinlock_wait_cycles":2304,"thread_start_us":117,"threads_started":1}
I20260619 15:01:30.019170 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.007235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.10.71.190:45827)
0619 15:01:30.007565 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.007581 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.007852 (+ 271us) server_negotiation.cc:407] Connection header received
0619 15:01:30.008814 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.008818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.008984 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.009133 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.009866 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.010790 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.011921 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.012244 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.012439 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.015067 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.015071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.015079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.015098 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.016639 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.017145 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.017149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.017150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.017231 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.017576 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.017578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.017579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.017855 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.017939 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.018338 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.018494 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
I20260619 15:01:30.021354 402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:30.021911 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.021662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:36956)
0619 15:01:30.021792 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.021866 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 15:01:30.022063 351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
I20260619 15:01:30.022763 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:36968)
0619 15:01:30.022668 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.022706 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 15:01:30.022939 402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:37857: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab.
Entry for principal kudu/127.10.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab.
I20260619 15:01:30.026160 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:37857
--webserver_interface=127.10.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab
--principal=kudu/127.10.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:45827,127.10.71.189:37857,127.10.71.188:34785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:01:30.072230 402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } attempt: 1
I20260619 15:01:30.072850 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.072651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:36970)
0619 15:01:30.072757 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.072802 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 15:01:30.073633 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.073368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:36986)
0619 15:01:30.073540 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.073583 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 15:01:30.073762 402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:37857: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
W20260619 15:01:30.125401 409 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:01:30.125648 409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:30.125708 409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:30.125792 409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:30.129199 409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:30.129266 409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:30.129300 409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:30.129354 409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:30.129388 409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:30.133627 409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:45827,127.10.71.189:37857,127.10.71.188:34785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab
--principal=kudu/127.10.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:30.134492 409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:30.135316 409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:01:30.141898 402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } attempt: 2
W20260619 15:01:30.142402 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:30.142496 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:30.142407 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:30.142782 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.142363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:36992)
0619 15:01:30.142657 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.142714 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260619 15:01:30.142976 409 server_base.cc:1034] running on GCE node
I20260619 15:01:30.143414 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.143252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:36994)
0619 15:01:30.143339 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.143375 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 15:01:30.143517 409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260619 15:01:30.143590 402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:37857: Network error: Client connection negotiation failed: client connection to 127.10.71.189:37857: connect: Connection refused (error 111)
I20260619 15:01:30.144513 409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:30.145684 409 hybrid_clock.cc:648] HybridClock initialized: now 1781881290145668 us; error 27 us; skew 500 ppm
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:30.147960 409 init.cc:377] Logged in from keytab as kudu/127.10.71.189@KRBTEST.COM (short username kudu)
I20260619 15:01:30.148715 409 webserver.cc:460] Webserver started at http://127.10.71.189:36893/ using document root <none> and password file <none>
I20260619 15:01:30.149103 409 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:30.149150 409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:30.149273 409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:30.150743 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data/instance:
uuid: "ac7f879df4ef4db79ff1239f2b27f6b0"
format_stamp: "Formatted at 2026-06-19 15:01:30 on dist-test-slave-3qdh"
server_key: "931a2b1dbcd36201c83a1aad760143c3"
server_key_iv: "2601e358aef255c1108f44f392167f3f"
server_key_version: "encryptionkey@0"
I20260619 15:01:30.151149 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal/instance:
uuid: "ac7f879df4ef4db79ff1239f2b27f6b0"
format_stamp: "Formatted at 2026-06-19 15:01:30 on dist-test-slave-3qdh"
server_key: "931a2b1dbcd36201c83a1aad760143c3"
server_key_iv: "2601e358aef255c1108f44f392167f3f"
server_key_version: "encryptionkey@0"
I20260619 15:01:30.154065 409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:30.155921 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:30.156956 409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:30.157037 409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal
uuid: "ac7f879df4ef4db79ff1239f2b27f6b0"
format_stamp: "Formatted at 2026-06-19 15:01:30 on dist-test-slave-3qdh"
server_key: "931a2b1dbcd36201c83a1aad760143c3"
server_key_iv: "2601e358aef255c1108f44f392167f3f"
server_key_version: "encryptionkey@0"
I20260619 15:01:30.157102 409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:30.174502 409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:30.176998 409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:30.177114 409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:30.183117 409 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.189:37857
I20260619 15:01:30.183382 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.189:37857 every 8 connection(s)
I20260619 15:01:30.183948 409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
I20260619 15:01:30.186187 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:30.189095 477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:30.190840 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 409
I20260619 15:01:30.190948 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-1/wal/instance
I20260619 15:01:30.191217 10526 external_mini_cluster.cc:1428] Setting key b930013796f9482be21030875c2b69e9
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.71.188@KRBTEST.COM; defaulting to no policy
I20260619 15:01:30.201284 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.190532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.10.71.190:45827)
0619 15:01:30.190633 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.190637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.191402 (+ 765us) server_negotiation.cc:407] Connection header received
0619 15:01:30.192165 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.192171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.192228 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.192335 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.193695 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.194551 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.195547 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.195951 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.196090 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.198144 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.198149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.198152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.198181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.199580 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.200019 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.200025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.200029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.200117 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.200454 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.200457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.200458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.200616 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.200685 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.201078 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.201167 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.188@KRBTEST.COM" created.
I20260619 15:01:30.201936 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.190825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45827 (local address 127.0.0.1:41252)
0619 15:01:30.191272 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.191312 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.191981 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.192487 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.192497 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.192911 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.193545 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.193565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.194679 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.194682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.195438 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.195445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.196102 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.196105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.196225 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.196780 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.196793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.197983 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.199677 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.199681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.199690 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.199908 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.200196 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.200198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.200199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.200380 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.200702 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.200706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.200884 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.201012 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.201211 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":147,"threads_started":1}
I20260619 15:01:30.203210 477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 15:01:30.211575 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.203636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:37006)
0619 15:01:30.203740 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.203755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.203830 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.204374 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.204380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.204575 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.204717 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.204728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.205899 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.205901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.206350 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.206357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.207163 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.207166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.207253 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.207690 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.207701 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.208764 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.210294 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.210297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.210298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.210498 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.210801 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.210803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.210804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.210870 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.211234 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.211236 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.211303 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.211380 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.211463 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260619 15:01:30.211719 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.203756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.10.71.189:37857)
0619 15:01:30.204011 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.204031 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.204053 (+ 22us) server_negotiation.cc:407] Connection header received
0619 15:01:30.204084 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.204088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.204209 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.204345 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.204858 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.205761 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.206475 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.207053 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.207161 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.208893 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.208900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.208908 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.208925 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.210208 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.210591 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.210599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.210604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.210693 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.210927 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.210929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.210930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.211164 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.211255 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.211430 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.211530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260619 15:01:30.213193 477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:30.213836 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.213606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46588)
0619 15:01:30.213710 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.213768 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 15:01:30.214040 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab.
Entry for principal kudu/127.10.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab.
I20260619 15:01:30.214792 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.214565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46598)
0619 15:01:30.214689 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.214739 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 15:01:30.214977 477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:34785: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
I20260619 15:01:30.215466 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:34785
--webserver_interface=127.10.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab
--principal=kudu/127.10.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:45827,127.10.71.189:37857,127.10.71.188:34785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:01:30.234740 402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } attempt: 3
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 15:01:30.242552 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.235252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.10.71.189:37857)
0619 15:01:30.235360 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.235365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.235379 (+ 14us) server_negotiation.cc:407] Connection header received
0619 15:01:30.235524 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.235527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.235566 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.235642 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.235985 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.236508 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.237488 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.237693 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.237805 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.239577 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.239582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.239584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.239609 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.241064 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.241518 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.241520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.241521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.241582 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.241867 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.241869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.241870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.242008 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.242071 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.242388 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.242461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 15:01:30.242671 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.235168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:37022)
0619 15:01:30.235324 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.235338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.235425 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.235656 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.235658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.235802 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.235885 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.235890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.236599 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.236605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.237382 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.237390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.237807 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.237810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.237906 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.238333 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.238343 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.239452 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.241170 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.241174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.241176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.241422 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.241668 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.241671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.241673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.241772 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.242096 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.242099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.242219 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.242357 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.242510 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 15:01:30.243742 402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:30.244431 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.244132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46614)
0619 15:01:30.244264 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.244357 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 15:01:30.245146 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.244898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46616)
0619 15:01:30.245034 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.245086 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 15:01:30.245292 402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:34785: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
I20260619 15:01:30.264202 477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } attempt: 1
I20260619 15:01:30.264849 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.264622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46626)
0619 15:01:30.264757 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.264800 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 15:01:30.265448 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.265277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46638)
0619 15:01:30.265375 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.265409 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 15:01:30.265591 477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:34785: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
I20260619 15:01:30.276458 402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } attempt: 1
I20260619 15:01:30.277004 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.276818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46640)
0619 15:01:30.276918 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.276958 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 15:01:30.277666 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.277449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46646)
0619 15:01:30.277572 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.277623 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 15:01:30.277878 402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:34785: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
W20260619 15:01:30.317057 484 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:01:30.317348 484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:30.317420 484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:30.317519 484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:30.321457 484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:30.321533 484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:30.321584 484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:30.321666 484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:30.321707 484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:30.326547 484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:45827,127.10.71.189:37857,127.10.71.188:34785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab
--principal=kudu/127.10.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:34785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:30.327438 484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:30.328392 484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 15:01:30.333783 477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } attempt: 2
I20260619 15:01:30.334583 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.334309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46656)
0619 15:01:30.334450 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.334510 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 15:01:30.335299 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.335063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46664)
0619 15:01:30.335164 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.335223 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 15:01:30.335469 477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:34785: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
W20260619 15:01:30.336280 490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:30.336375 492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:30.336360 489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:30.336689 484 server_base.cc:1034] running on GCE node
I20260619 15:01:30.337163 484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:30.338130 484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:30.339286 484 hybrid_clock.cc:648] HybridClock initialized: now 1781881290339268 us; error 25 us; skew 500 ppm
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:30.342121 484 init.cc:377] Logged in from keytab as kudu/127.10.71.188@KRBTEST.COM (short username kudu)
I20260619 15:01:30.342986 484 webserver.cc:460] Webserver started at http://127.10.71.188:45357/ using document root <none> and password file <none>
I20260619 15:01:30.343539 484 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:30.343633 484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:30.343854 484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:30.345732 484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data/instance:
uuid: "731d0db8924a4742bed9684a6692ea93"
format_stamp: "Formatted at 2026-06-19 15:01:30 on dist-test-slave-3qdh"
server_key: "311c57527db9b500f33fa4816ee122cf"
server_key_iv: "bc37e2f7f2b2ff49558ea04b5133689b"
server_key_version: "encryptionkey@0"
I20260619 15:01:30.346243 484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal/instance:
uuid: "731d0db8924a4742bed9684a6692ea93"
format_stamp: "Formatted at 2026-06-19 15:01:30 on dist-test-slave-3qdh"
server_key: "311c57527db9b500f33fa4816ee122cf"
server_key_iv: "bc37e2f7f2b2ff49558ea04b5133689b"
server_key_version: "encryptionkey@0"
I20260619 15:01:30.349890 484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:30.352070 499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:30.353032 402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } attempt: 2
I20260619 15:01:30.353250 484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 15:01:30.353369 484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal
uuid: "731d0db8924a4742bed9684a6692ea93"
format_stamp: "Formatted at 2026-06-19 15:01:30 on dist-test-slave-3qdh"
server_key: "311c57527db9b500f33fa4816ee122cf"
server_key_iv: "bc37e2f7f2b2ff49558ea04b5133689b"
server_key_version: "encryptionkey@0"
I20260619 15:01:30.353492 484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:30.353809 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.353459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46670)
0619 15:01:30.353641 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.353727 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 15:01:30.354532 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.354329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46678)
0619 15:01:30.354428 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.354468 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 15:01:30.354743 402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:34785: Network error: Client connection negotiation failed: client connection to 127.10.71.188:34785: connect: Connection refused (error 111)
I20260619 15:01:30.363063 484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:30.365589 484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:30.365789 484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:30.372459 484 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.188:34785
I20260619 15:01:30.372728 551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.188:34785 every 8 connection(s)
I20260619 15:01:30.373313 484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
I20260619 15:01:30.375696 552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:30.379123 552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:30.380322 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 484
I20260619 15:01:30.380450 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781881028756249-10526-0/minicluster-data/master-2/wal/instance
I20260619 15:01:30.380765 10526 external_mini_cluster.cc:1428] Setting key 1b367d7857939f2ad9158eab44cb08e5
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 15:01:30.391844 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.380775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.10.71.190:45827)
0619 15:01:30.380925 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.380931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.381859 (+ 928us) server_negotiation.cc:407] Connection header received
0619 15:01:30.382841 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.382845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.382896 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.382987 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.383745 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.384246 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.385307 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.385545 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.385650 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.388322 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.388326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.388329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.388354 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.389952 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.390413 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.390418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.390420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.390508 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.390826 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.390830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.390832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.391022 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.391125 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.391578 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.391685 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 15:01:30.392680 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.381191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:45827 (local address 127.0.0.1:41266)
0619 15:01:30.381686 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.381758 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.382643 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.383039 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.383044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.383354 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.383607 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.383619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.384379 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.384384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.385185 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.385194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.385676 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.385680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.385847 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.386401 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.386416 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.388154 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.390061 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.390068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.390076 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.390311 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.390598 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.390602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.390604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.390731 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.391124 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.391130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.391345 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.391517 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.391785 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":72,"thread_start_us":150,"threads_started":1}
I20260619 15:01:30.392680 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.381421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.10.71.188:34785)
0619 15:01:30.381837 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.381861 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.381988 (+ 127us) server_negotiation.cc:407] Connection header received
0619 15:01:30.382663 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.382684 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.382852 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.383067 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.383942 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.384862 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.385696 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.386078 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.386280 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.388978 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.388985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.388995 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.389025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.390488 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.391069 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.391076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.391078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.391169 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.391472 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.391477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.391479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.391707 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.391916 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.392164 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.392327 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":128,"threads_started":1}
I20260619 15:01:30.393834 552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:30.401460 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.394270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:37857 (local address 127.0.0.1:37034)
0619 15:01:30.394481 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.394495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.394578 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.394812 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.394814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.394966 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.395040 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.395044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.395698 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.395704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.396239 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.396246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.396736 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.396739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.396835 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.397258 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.397272 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.398366 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.400044 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.400049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.400051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.400240 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.400580 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.400584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.400586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.400663 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.401018 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.401021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.401108 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.401210 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.401329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260619 15:01:30.401505 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.394376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.10.71.189:37857)
0619 15:01:30.394477 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.394482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.394543 (+ 61us) server_negotiation.cc:407] Connection header received
0619 15:01:30.394652 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.394655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.394695 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.394803 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.395151 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.395604 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.396384 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.396627 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.396737 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.398493 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.398496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.398498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.398526 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.399930 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.400375 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.400378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.400380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.400487 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.400752 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.400757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.400759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.400920 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.401027 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.401269 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.401364 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 15:01:30.402285 552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:30.403654 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.394854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.10.71.190:45827)
0619 15:01:30.394954 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.394958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.395037 (+ 79us) server_negotiation.cc:407] Connection header received
0619 15:01:30.395160 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.395163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.395208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.395305 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.395845 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.396442 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.397214 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.397438 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.397556 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.400083 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.400094 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.400096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.400117 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.401593 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.402084 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.402090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.402092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.402178 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.402815 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.402819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.402820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.403023 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.403234 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.403354 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.403449 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 15:01:30.410550 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.402731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46688)
0619 15:01:30.402842 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.402855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.402918 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.403159 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.403163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.403308 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.403383 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.403389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.404315 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.404319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.405031 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.405039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.405572 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.405576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.405648 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.406169 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.406179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.407246 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.409087 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.409090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.409092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.409267 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.409568 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.409570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.409572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.409649 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.410030 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.410035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.410130 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.410257 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.410372 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 15:01:30.410578 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.402852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.10.71.188:34785)
0619 15:01:30.402951 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.402968 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.402979 (+ 11us) server_negotiation.cc:407] Connection header received
0619 15:01:30.403006 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.403008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.403038 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.403129 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.403519 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.404186 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.405249 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.405446 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.405535 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.407363 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.407365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.407367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.407384 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.408981 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.409358 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.409361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.409363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.409431 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.409742 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.409746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.409750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.409931 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.410007 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.410299 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.410447 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":42}
I20260619 15:01:30.413103 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.404937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.10.71.189:37857)
0619 15:01:30.405021 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.405024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.405038 (+ 14us) server_negotiation.cc:407] Connection header received
0619 15:01:30.405097 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.405099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.405127 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.405180 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.405727 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.406362 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.407153 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.407434 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.407502 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.409909 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.409913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.409915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.409935 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.411455 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.411870 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.411876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.411877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.411956 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.412229 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.412232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.412234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.412456 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.412693 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.412841 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.412948 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 15:01:30.413635 552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: Bootstrap starting.
I20260619 15:01:30.416208 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:30.416329 552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:30.416972 552 log.cc:824] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:30.417364 376 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "55535589ac8b42e6a8d5093343a1a353"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41270
I20260619 15:01:30.418628 552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: No bootstrap required, opened a new log
I20260619 15:01:30.422104 552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.422297 552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731d0db8924a4742bed9684a6692ea93, State: Initialized, Role: FOLLOWER
I20260619 15:01:30.422596 552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.423053 558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.423154 558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.423816 552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:30.426632 477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } attempt: 3
W20260619 15:01:30.428267 569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:30.428360 569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:30.427074 484 master_runner.cc:186] Error getting master registration for 127.10.71.190:45827: OK, instance_id { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" instance_seqno: 1781881289996766 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:01:30.429705 484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 15:01:30.434988 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.427021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46690)
0619 15:01:30.427169 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.427182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.427248 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.427605 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.427607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.427783 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.427864 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.427872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.428900 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.428905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.429667 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.429677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.430184 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.430188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.430281 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.430675 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.430687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.431764 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.433457 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.433460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.433462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.433718 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.434007 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.434010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.434012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.434094 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.434518 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.434521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.434606 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.434718 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.434830 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 15:01:30.435042 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.427066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.10.71.188:34785)
0619 15:01:30.427365 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.427368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.427379 (+ 11us) server_negotiation.cc:407] Connection header received
0619 15:01:30.427407 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.427409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.427441 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.427513 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.427985 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.428700 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.429790 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.430060 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.430152 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.431884 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.431887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.431890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.431910 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.433345 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.433820 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.433824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.433826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.433909 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.434190 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.434193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.434195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.434420 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.434529 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.434784 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.434896 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260619 15:01:30.437666 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: Bootstrap starting.
I20260619 15:01:30.440022 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:30.440778 477 log.cc:824] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:30.442463 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: No bootstrap required, opened a new log
I20260619 15:01:30.446010 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.446189 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7f879df4ef4db79ff1239f2b27f6b0, State: Initialized, Role: FOLLOWER
I20260619 15:01:30.446444 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.446846 572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.446965 572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.447520 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 15:01:30.451745 583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:30.451849 583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:30.450594 409 master_runner.cc:186] Error getting master registration for 127.10.71.190:45827: OK, instance_id { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" instance_seqno: 1781881289996766 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:01:30.453369 409 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:01:30.453838 402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } attempt: 3
Jun 19 15:01:30 dist-test-slave-3qdh krb5kdc[321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 15:01:30.461586 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.454274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.10.71.188:34785)
0619 15:01:30.454361 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.454364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.454540 (+ 176us) server_negotiation.cc:407] Connection header received
0619 15:01:30.454575 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.454578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.454615 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.454684 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.455098 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.455500 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.456450 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.456681 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.456780 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.458636 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.458639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.458641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.458660 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.460043 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.460473 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.460477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.460479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.460551 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.460825 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.460828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.460829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.461007 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.461099 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.461368 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.461457 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 15:01:30.461586 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.454198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:34785 (local address 127.0.0.1:46704)
0619 15:01:30.454351 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:30.454365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:30.454441 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:30.454733 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:30.454736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:30.454918 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:30.455014 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.455020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.455603 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.455607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.456352 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:30.456362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.456781 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.456784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:30.456858 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.457253 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:30.457264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:30.458528 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:30.460125 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.460128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.460130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.460373 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.460627 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:30.460630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:30.460631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:30.460752 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:30.461108 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:30.461114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:30.461198 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 15:01:30.461323 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 15:01:30.461446 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 15:01:30.464437 402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: Bootstrap starting.
I20260619 15:01:30.466816 402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:30.467556 402 log.cc:824] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:30.469383 402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: No bootstrap required, opened a new log
I20260619 15:01:30.471899 402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.472080 402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55535589ac8b42e6a8d5093343a1a353, State: Initialized, Role: FOLLOWER
I20260619 15:01:30.472368 402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.472777 586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.473002 586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.473497 402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:30.473565 376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 15:01:30.473649 376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:30.474851 376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.475666 376 leader_election.cc:288] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [CANDIDATE]: Term 1 election: Requested vote from peers ac7f879df4ef4db79ff1239f2b27f6b0 (127.10.71.189:37857), 731d0db8924a4742bed9684a6692ea93 (127.10.71.188:34785)
I20260619 15:01:30.476460 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55535589ac8b42e6a8d5093343a1a353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0"
I20260619 15:01:30.476675 451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:30.477944 451 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55535589ac8b42e6a8d5093343a1a353 in term 1.
I20260619 15:01:30.478489 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55535589ac8b42e6a8d5093343a1a353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "731d0db8924a4742bed9684a6692ea93"
I20260619 15:01:30.478678 526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:30.479801 526 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55535589ac8b42e6a8d5093343a1a353 in term 1.
W20260619 15:01:30.481427 597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:30.481554 597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:30.481720 351 leader_election.cc:302] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55535589ac8b42e6a8d5093343a1a353, ac7f879df4ef4db79ff1239f2b27f6b0; no voters:
I20260619 15:01:30.481937 586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:30.482122 586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [term 1 LEADER]: Becoming Leader. State: Replica: 55535589ac8b42e6a8d5093343a1a353, State: Running, Role: LEADER
I20260619 15:01:30.482416 586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } }
I20260619 15:01:30.483997 586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55535589ac8b42e6a8d5093343a1a353. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.484099 586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:30.484612 605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:30.485512 605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:30.486331 334 master_runner.cc:263] No leader master found from master 55535589ac8b42e6a8d5093343a1a353
I20260619 15:01:30.486388 334 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:01:30.490129 526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [term 1 FOLLOWER]: Refusing update from remote peer 55535589ac8b42e6a8d5093343a1a353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:30.490736 586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 15:01:30.493643 355 tablet.cc:2369] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:30.494590 558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55535589ac8b42e6a8d5093343a1a353. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.494712 558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.495026 451 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [term 1 FOLLOWER]: Refusing update from remote peer 55535589ac8b42e6a8d5093343a1a353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:30.495579 586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:30.498799 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.486739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.10.71.188:34785)
0619 15:01:30.486829 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.486833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.487498 (+ 665us) server_negotiation.cc:407] Connection header received
0619 15:01:30.489450 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.489453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.489508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.489559 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.490628 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.491277 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.492681 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.492990 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.493062 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.495104 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.495107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.495110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.495134 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.496953 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.497547 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.497551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.497553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.497630 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.497950 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.497954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.497955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.498161 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.498245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.498585 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.498678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 15:01:30.499541 572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55535589ac8b42e6a8d5093343a1a353. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.499657 572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.504235 572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.504387 572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.508219 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.487403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.10.71.190:45827)
0619 15:01:30.487501 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.487505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.500071 (+ 12566us) server_negotiation.cc:407] Connection header received
0619 15:01:30.500231 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.500234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.500276 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.500355 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.501087 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.501698 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.502657 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.502919 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.502997 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.504774 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.504778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.504780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.504801 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.506365 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.506806 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.506811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.506817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.506902 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.507230 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.507235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.507239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.507483 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.507586 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.507947 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.508063 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 15:01:30.509831 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:30.487691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.10.71.189:37857)
0619 15:01:30.487806 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.487810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.501244 (+ 13434us) server_negotiation.cc:407] Connection header received
0619 15:01:30.501331 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.501333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.501373 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.501444 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.502290 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.502930 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.504172 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:30.504524 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:30.504594 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:30.506343 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:30.506346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:30.506348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:30.506369 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:30.507982 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.508431 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.508460 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.508468 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.508585 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:30.508890 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:30.508893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:30.508895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:30.509096 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:30.509197 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:30.509565 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 15:01:30.509674 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 15:01:30.510035 612 mvcc.cc:205] Tried to move back new op lower bound from 7298585765842407424 to 7298585765818609664. Current Snapshot: MvccSnapshot[applied={T|T < 7298585765842407424}]
I20260619 15:01:30.509848 607 mvcc.cc:205] Tried to move back new op lower bound from 7298585765842407424 to 7298585765818609664. Current Snapshot: MvccSnapshot[applied={T|T < 7298585765842407424}]
I20260619 15:01:30.511324 586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.511436 586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:30.511463 603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.511529 603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353 [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:30.512524 558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55535589ac8b42e6a8d5093343a1a353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55535589ac8b42e6a8d5093343a1a353" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 45827 } } peers { permanent_uuid: "ac7f879df4ef4db79ff1239f2b27f6b0" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 37857 } } peers { permanent_uuid: "731d0db8924a4742bed9684a6692ea93" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 34785 } } }
I20260619 15:01:30.512616 558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:30.513161 605 catalog_manager.cc:1347] Generated new cluster ID: c25a9e6d349243f682f6ebe51addf822
I20260619 15:01:30.513214 605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:30.516371 610 mvcc.cc:205] Tried to move back new op lower bound from 7298585765842407424 to 7298585765818609664. Current Snapshot: MvccSnapshot[applied={T|T < 7298585765842407424 or (T in {7298585765842407424})}]
I20260619 15:01:30.526154 605 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:30.527173 605 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:30.538230 605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55535589ac8b42e6a8d5093343a1a353: Generated new TSK 0
I20260619 15:01:30.538885 605 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:01:30.569103 555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.565957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.10.71.188:34785)
0619 15:01:30.566086 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.566091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.567753 (+ 1662us) server_negotiation.cc:407] Connection header received
0619 15:01:30.568005 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.568044 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.568114 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.568173 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.568978 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46732: BlockingRecv error: recv got EOF from 127.0.0.1:46732 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 15:01:30.570737 405 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.566779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.10.71.190:45827)
0619 15:01:30.566905 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.566921 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.569479 (+ 2558us) server_negotiation.cc:407] Connection header received
0619 15:01:30.570050 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.570053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.570095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.570139 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.570630 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41286: BlockingRecv error: recv got EOF from 127.0.0.1:41286 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 15:01:30.571593 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.567407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.10.71.189:37857)
0619 15:01:30.567517 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.567522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.569794 (+ 2272us) server_negotiation.cc:407] Connection header received
0619 15:01:30.570978 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.570981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.571027 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.571074 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.571493 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37056: BlockingRecv error: recv got EOF from 127.0.0.1:37056 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 15:01:30.575675 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.574319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.10.71.189:37857)
0619 15:01:30.574411 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.574415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.574857 (+ 442us) server_negotiation.cc:407] Connection header received
0619 15:01:30.575065 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.575067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.575139 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.575181 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.575599 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37066: BlockingRecv error: recv got EOF from 127.0.0.1:37066 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 15:01:30.580072 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.578831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.10.71.189:37857)
0619 15:01:30.578919 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.578922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.579398 (+ 476us) server_negotiation.cc:407] Connection header received
0619 15:01:30.579536 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.579539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.579575 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.579612 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.580003 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37082: BlockingRecv error: recv got EOF from 127.0.0.1:37082 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 15:01:30.585239 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.583844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.10.71.189:37857)
0619 15:01:30.583937 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.583941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.584495 (+ 554us) server_negotiation.cc:407] Connection header received
0619 15:01:30.584675 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.584678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.584719 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.584770 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.585174 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37092: BlockingRecv error: recv got EOF from 127.0.0.1:37092 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 15:01:30.592357 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.591077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.10.71.189:37857)
0619 15:01:30.591173 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.591176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.591576 (+ 400us) server_negotiation.cc:407] Connection header received
0619 15:01:30.591717 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.591720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.591759 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.591798 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.592236 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37098: BlockingRecv error: recv got EOF from 127.0.0.1:37098 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 15:01:30.603539 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.602227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.10.71.189:37857)
0619 15:01:30.602309 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.602313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.602741 (+ 428us) server_negotiation.cc:407] Connection header received
0619 15:01:30.602902 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.602905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.602962 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.603009 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.603441 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37106: BlockingRecv error: recv got EOF from 127.0.0.1:37106 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 15:01:30.623322 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.621789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.10.71.189:37857)
0619 15:01:30.621896 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.621901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.622407 (+ 506us) server_negotiation.cc:407] Connection header received
0619 15:01:30.622605 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.622609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.622677 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.622722 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.623208 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37114: BlockingRecv error: recv got EOF from 127.0.0.1:37114 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 15:01:30.658507 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.657475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.10.71.189:37857)
0619 15:01:30.657624 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.657628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.657754 (+ 126us) server_negotiation.cc:407] Connection header received
0619 15:01:30.657881 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.657884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.657937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.658030 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.658409 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37122: BlockingRecv error: recv got EOF from 127.0.0.1:37122 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 15:01:30.725494 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.724500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.10.71.189:37857)
0619 15:01:30.724621 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.724624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.724802 (+ 178us) server_negotiation.cc:407] Connection header received
0619 15:01:30.724902 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.724904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.724988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.725068 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.725403 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37138: BlockingRecv error: recv got EOF from 127.0.0.1:37138 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 15:01:30.856366 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:30.855370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.10.71.189:37857)
0619 15:01:30.855503 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:30.855507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:30.855680 (+ 173us) server_negotiation.cc:407] Connection header received
0619 15:01:30.855787 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:30.855789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:30.855832 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:30.855897 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:30.856225 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37144: BlockingRecv error: recv got EOF from 127.0.0.1:37144 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 15:01:31.115257 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:31.114303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.10.71.189:37857)
0619 15:01:31.114448 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:31.114452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:31.114470 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:01:31.114613 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:31.114615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:31.114661 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:31.114749 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:31.115147 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37152: BlockingRecv error: recv got EOF from 127.0.0.1:37152 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 15:01:31.429347 569 catalog_manager.cc:1259] Loaded cluster ID: c25a9e6d349243f682f6ebe51addf822
I20260619 15:01:31.429404 569 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: loading cluster ID for follower catalog manager: success
I20260619 15:01:31.430917 569 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: acquiring CA information for follower catalog manager: success
I20260619 15:01:31.431921 569 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 731d0db8924a4742bed9684a6692ea93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:01:31.452893 583 catalog_manager.cc:1259] Loaded cluster ID: c25a9e6d349243f682f6ebe51addf822
I20260619 15:01:31.452948 583 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: loading cluster ID for follower catalog manager: success
I20260619 15:01:31.454463 583 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: acquiring CA information for follower catalog manager: success
I20260619 15:01:31.455478 583 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac7f879df4ef4db79ff1239f2b27f6b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:01:31.632472 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:31.629186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.10.71.189:37857)
0619 15:01:31.629386 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:31.629389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:31.629452 (+ 63us) server_negotiation.cc:407] Connection header received
0619 15:01:31.629579 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:31.629581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:31.629698 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:31.629784 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:31.629976 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.630410 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.631169 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.631374 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.631445 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:31.631695 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:31.632133 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:31.632216 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:31.632258 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 15:01:31.632351 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260619 15:01:31.637982 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:31.635135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.10.71.188:34785)
0619 15:01:31.635327 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:31.635330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:31.635368 (+ 38us) server_negotiation.cc:407] Connection header received
0619 15:01:31.635497 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:31.635499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:31.635595 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:31.635674 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:31.635833 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.636251 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.636926 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.637112 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.637177 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:31.637372 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:31.637666 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:31.637741 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:31.637784 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 15:01:31.637821 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":52,"threads_started":1}
I20260619 15:01:31.643659 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:31.640447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.10.71.188:34785)
0619 15:01:31.640532 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:31.640537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:31.640796 (+ 259us) server_negotiation.cc:407] Connection header received
0619 15:01:31.640920 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:31.640924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:31.640968 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:31.640999 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:31.641405 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.641885 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.642657 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.642915 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.643018 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:31.643232 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:31.643287 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:31.643345 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:31.643480 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 15:01:31.643537 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 15:01:31.644064 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:31.640797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.10.71.189:37857)
0619 15:01:31.640912 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:31.640916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:31.641068 (+ 152us) server_negotiation.cc:407] Connection header received
0619 15:01:31.641211 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:31.641213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:31.641256 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:31.641291 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:31.641598 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.642031 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.642964 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.643275 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.643375 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:31.643591 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:31.643669 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:31.643748 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:31.643896 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 15:01:31.643948 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 15:01:31.645303 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:31.640337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.10.71.190:45827)
0619 15:01:31.640550 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:31.640554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:31.640735 (+ 181us) server_negotiation.cc:407] Connection header received
0619 15:01:31.640885 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:31.640889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:31.641029 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:31.641152 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:31.641983 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.642656 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.643756 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:31.644125 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:31.644225 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:31.644605 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:31.644891 (+ 286us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:31.644981 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:31.645141 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 15:01:31.645201 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260619 15:01:31.647639 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 334
I20260619 15:01:31.653491 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 409
I20260619 15:01:31.658551 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 484
2026-06-19T15:01:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2868 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 15:01:31 dist-test-slave-3qdh krb5kdc[702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 15:01:31 dist-test-slave-3qdh krb5kdc[702](info): set up 2 sockets
Jun 19 15:01:31 dist-test-slave-3qdh krb5kdc[702](info): commencing operation
krb5kdc: starting...
W20260619 15:01:32.608492 10526 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/test-user.keytab.
Jun 19 15:01:32 dist-test-slave-3qdh krb5kdc[702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T15:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T15:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
Entry for principal kudu/127.10.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab.
I20260619 15:01:32.711968 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:43209
--webserver_interface=127.10.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:43209,127.10.71.189:45787,127.10.71.188:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 15:01:32.806481 715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:01:32.806687 715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:32.806741 715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:32.806809 715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:32.810362 715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:32.810442 715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:32.810473 715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:32.810501 715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:32.810524 715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:32.817917 715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:43209,127.10.71.189:45787,127.10.71.188:37099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.190.keytab
--principal=kudu/127.10.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.190:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:32.818821 715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:32.819667 715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:32.826547 721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:32.826627 723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:32.826633 720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:32.826705 715 server_base.cc:1034] running on GCE node
I20260619 15:01:32.827226 715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:32.828049 715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:32.829190 715 hybrid_clock.cc:648] HybridClock initialized: now 1781881292829162 us; error 40 us; skew 500 ppm
Jun 19 15:01:32 dist-test-slave-3qdh krb5kdc[702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:32.835839 715 init.cc:377] Logged in from keytab as kudu/127.10.71.190@KRBTEST.COM (short username kudu)
I20260619 15:01:32.836575 715 webserver.cc:460] Webserver started at http://127.10.71.190:42403/ using document root <none> and password file <none>
I20260619 15:01:32.836973 715 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:32.837021 715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:32.837152 715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:32.838709 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data/instance:
uuid: "3642be3a1ae3484988cbc9a6fe6570ac"
format_stamp: "Formatted at 2026-06-19 15:01:32 on dist-test-slave-3qdh"
server_key: "d68bf3de2059d1ca36053c35469cd4cf"
server_key_iv: "3b7f9c99bb718ad5284c44cdf2bfab02"
server_key_version: "encryptionkey@0"
I20260619 15:01:32.839155 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal/instance:
uuid: "3642be3a1ae3484988cbc9a6fe6570ac"
format_stamp: "Formatted at 2026-06-19 15:01:32 on dist-test-slave-3qdh"
server_key: "d68bf3de2059d1ca36053c35469cd4cf"
server_key_iv: "3b7f9c99bb718ad5284c44cdf2bfab02"
server_key_version: "encryptionkey@0"
I20260619 15:01:32.842082 715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 15:01:32.844000 730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:32.845121 715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:32.845228 715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal
uuid: "3642be3a1ae3484988cbc9a6fe6570ac"
format_stamp: "Formatted at 2026-06-19 15:01:32 on dist-test-slave-3qdh"
server_key: "d68bf3de2059d1ca36053c35469cd4cf"
server_key_iv: "3b7f9c99bb718ad5284c44cdf2bfab02"
server_key_version: "encryptionkey@0"
I20260619 15:01:32.845312 715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:32.861575 715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:32.864113 715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:32.864295 715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:32.869861 715 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.190:43209
I20260619 15:01:32.870127 782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.190:43209 every 8 connection(s)
I20260619 15:01:32.870651 715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/data/info.pb
I20260619 15:01:32.872952 783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:32.876031 783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:32.876413 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 715
I20260619 15:01:32.876533 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-0/wal/instance
I20260619 15:01:32.876758 10526 external_mini_cluster.cc:1428] Setting key fca1d9f40a73fbe01c2f161f6cb6fee5
WARNING: no policy specified for kudu/127.10.71.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.189@KRBTEST.COM" created.
Jun 19 15:01:32 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:32.892411 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:32.878008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:43209 (local address 127.0.0.1:46078)
0619 15:01:32.878507 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:32.878546 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:32.879206 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:32.879764 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:32.879771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:32.880188 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:32.880568 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:32.880589 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:32.881584 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:32.881589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:32.882677 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:32.882691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:32.883220 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:32.883224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:32.883351 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:32.883822 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:32.883845 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:32.887843 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:32.889980 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:32.889987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:32.889992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:32.890187 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:32.890560 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:32.890582 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:32.890585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:32.890713 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:32.891367 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:32.891370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:32.891462 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 15:01:32.891576 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 15:01:32.891754 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"spinlock_wait_cycles":34048,"thread_start_us":129,"threads_started":1}
I20260619 15:01:32.892414 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:32.878007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.10.71.190:43209)
0619 15:01:32.878476 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:32.878491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:32.878644 (+ 153us) server_negotiation.cc:407] Connection header received
0619 15:01:32.879427 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:32.879431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:32.879583 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:32.879707 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:32.880732 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:32.881462 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:32.882790 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:32.883082 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:32.883177 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:32.887984 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:32.887988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:32.887996 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:32.888015 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:32.889881 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:32.890293 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:32.890301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:32.890305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:32.890469 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:32.890805 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:32.890810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:32.890813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:32.891260 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:32.891387 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:32.891682 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 15:01:32.891826 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":110,"threads_started":1}
I20260619 15:01:32.895152 783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:32.895715 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:32.895518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58170)
0619 15:01:32.895617 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:32.895672 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 15:01:32.895951 734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
I20260619 15:01:32.896713 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:32.896522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58174)
0619 15:01:32.896611 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:32.896654 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 15:01:32.896852 783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:45787: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab.
Entry for principal kudu/127.10.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab.
I20260619 15:01:32.900949 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:45787
--webserver_interface=127.10.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab
--principal=kudu/127.10.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:43209,127.10.71.189:45787,127.10.71.188:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:01:32.946033 783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } attempt: 1
I20260619 15:01:32.946611 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:32.946398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58190)
0619 15:01:32.946503 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:32.946546 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 15:01:32.947189 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:32.947010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58206)
0619 15:01:32.947099 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:32.947145 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 15:01:32.947328 783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:45787: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
W20260619 15:01:32.994632 790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:01:32.994911 790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:32.994984 790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:32.995055 790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:32.998458 790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:32.998521 790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:32.998551 790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:32.998584 790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:32.998613 790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:33.002795 790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:43209,127.10.71.189:45787,127.10.71.188:37099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.189.keytab
--principal=kudu/127.10.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.189:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:33.003549 790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:33.004462 790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:33.011206 798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:33.011251 796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:33.011217 795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:33.011255 790 server_base.cc:1034] running on GCE node
I20260619 15:01:33.011791 790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:33.012748 790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:33.013985 790 hybrid_clock.cc:648] HybridClock initialized: now 1781881293013975 us; error 26 us; skew 500 ppm
I20260619 15:01:33.015467 783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } attempt: 2
I20260619 15:01:33.016104 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.015844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58218)
0619 15:01:33.015968 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.016024 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:33.016660 790 init.cc:377] Logged in from keytab as kudu/127.10.71.189@KRBTEST.COM (short username kudu)
I20260619 15:01:33.016765 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.016534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58230)
0619 15:01:33.016644 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.016683 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 15:01:33.016932 783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.189:45787: Network error: Client connection negotiation failed: client connection to 127.10.71.189:45787: connect: Connection refused (error 111)
I20260619 15:01:33.017418 790 webserver.cc:460] Webserver started at http://127.10.71.189:34953/ using document root <none> and password file <none>
I20260619 15:01:33.017850 790 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:33.017925 790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:33.018113 790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:33.019750 790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data/instance:
uuid: "801ee8abb07548bc8cbe0226360c0cce"
format_stamp: "Formatted at 2026-06-19 15:01:33 on dist-test-slave-3qdh"
server_key: "dee1db82d42d8aa973fb3ca1e1495908"
server_key_iv: "74b94e25f900d65da79392f8a3ead2d8"
server_key_version: "encryptionkey@0"
I20260619 15:01:33.020205 790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal/instance:
uuid: "801ee8abb07548bc8cbe0226360c0cce"
format_stamp: "Formatted at 2026-06-19 15:01:33 on dist-test-slave-3qdh"
server_key: "dee1db82d42d8aa973fb3ca1e1495908"
server_key_iv: "74b94e25f900d65da79392f8a3ead2d8"
server_key_version: "encryptionkey@0"
I20260619 15:01:33.023043 790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 15:01:33.025166 805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:33.026101 790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:33.026219 790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal
uuid: "801ee8abb07548bc8cbe0226360c0cce"
format_stamp: "Formatted at 2026-06-19 15:01:33 on dist-test-slave-3qdh"
server_key: "dee1db82d42d8aa973fb3ca1e1495908"
server_key_iv: "74b94e25f900d65da79392f8a3ead2d8"
server_key_version: "encryptionkey@0"
I20260619 15:01:33.026312 790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:33.052239 790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:33.054939 790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:33.055110 790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:33.060678 790 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.189:45787
I20260619 15:01:33.060964 857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.189:45787 every 8 connection(s)
I20260619 15:01:33.061439 790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/data/info.pb
I20260619 15:01:33.063635 858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:33.065508 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 790
I20260619 15:01:33.065622 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-1/wal/instance
I20260619 15:01:33.065874 10526 external_mini_cluster.cc:1428] Setting key f4cbf1a8fe07a08359d1168bcb637322
I20260619 15:01:33.067356 858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.71.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.71.188@KRBTEST.COM" created.
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:33.079928 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.068913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.10.71.190:43209)
0619 15:01:33.069046 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.069052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.069661 (+ 609us) server_negotiation.cc:407] Connection header received
0619 15:01:33.070404 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.070408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.070467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.070564 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.071417 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.072165 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.073271 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.073483 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.073559 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.076104 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.076107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.076110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.076130 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.078251 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.078685 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.078689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.078691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.078763 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.079049 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.079052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.079053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.079200 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.079293 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.079694 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.079799 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 15:01:33.080570 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.069180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:43209 (local address 127.0.0.1:46090)
0619 15:01:33.069528 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.069569 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.070227 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.070675 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.070680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.070938 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.071287 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.071297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.072335 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.072341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.073170 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.073179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.073591 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.073594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.073713 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.074227 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.074241 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.075940 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.078358 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.078363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.078373 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.078595 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.078843 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.078847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.078848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.078962 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.079292 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.079297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.079485 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.079643 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.079881 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260619 15:01:33.081836 858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab.
Entry for principal kudu/127.10.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab.
I20260619 15:01:33.089359 10526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskn16cqy/build/debug/bin/kudu
/tmp/dist-test-taskn16cqy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:37099
--webserver_interface=127.10.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab
--principal=kudu/127.10.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.71.148:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.71.190:43209,127.10.71.189:45787,127.10.71.188:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 15:01:33.090368 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.082259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58246)
0619 15:01:33.082354 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.082363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.082413 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.082876 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.082878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.082979 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.083057 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.083061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.084091 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.084093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.084665 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.084673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.085293 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.085296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.085370 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.085743 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.085752 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.086855 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.088807 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.088810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.088811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.089036 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.089291 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.089293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.089295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.089392 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.089809 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.089814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.089912 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.090021 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.090154 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260619 15:01:33.090430 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.082311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.10.71.189:45787)
0619 15:01:33.082556 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.082569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.082584 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:01:33.082615 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.082618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.082787 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.082874 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.083135 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.083996 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.084762 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.085155 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.085268 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.087006 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.087013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.087021 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.087038 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.088694 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.089135 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.089139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.089140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.089209 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.089487 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.089491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.089492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.089715 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.089800 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.090066 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.090199 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260619 15:01:33.091763 858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:33.092406 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.092111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49724)
0619 15:01:33.092232 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.092329 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 15:01:33.092583 809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
I20260619 15:01:33.093362 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.093118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49736)
0619 15:01:33.093242 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.093297 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 15:01:33.093509 858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:37099: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
I20260619 15:01:33.108108 783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } attempt: 3
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 15:01:33.115576 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.108463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58248)
0619 15:01:33.108575 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.108588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.108672 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.108894 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.108896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.109021 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.109088 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.109092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.109933 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.109935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.110374 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.110380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.110876 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.110878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.110948 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.111352 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.111362 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.112624 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.114317 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.114319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.114320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.114498 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.114773 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.114775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.114776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.114843 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.115203 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.115205 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.115266 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.115370 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.115481 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 15:01:33.115593 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.108532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.10.71.189:45787)
0619 15:01:33.108629 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.108633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.108646 (+ 13us) server_negotiation.cc:407] Connection header received
0619 15:01:33.108753 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.108755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.108793 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.108875 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.109204 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.109841 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.110468 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.110765 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.110873 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.112778 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.112782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.112784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.112808 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.114217 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.114599 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.114602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.114604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.114686 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.114946 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.114948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.114950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.115127 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.115208 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.115394 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.115484 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 15:01:33.116677 783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:33.117985 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.117709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49750)
0619 15:01:33.117831 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.117901 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 15:01:33.118723 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.118435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49762)
0619 15:01:33.118604 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.118650 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 15:01:33.119165 783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:37099: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
I20260619 15:01:33.142810 858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } attempt: 1
I20260619 15:01:33.143312 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.143127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49764)
0619 15:01:33.143225 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.143266 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 15:01:33.143963 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.143766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49770)
0619 15:01:33.143873 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.143920 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 15:01:33.144088 858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:37099: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
I20260619 15:01:33.150307 783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } attempt: 1
I20260619 15:01:33.150813 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.150584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49778)
0619 15:01:33.150701 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.150752 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 15:01:33.151419 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.151232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49790)
0619 15:01:33.151323 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.151369 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 15:01:33.151547 783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:37099: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
W20260619 15:01:33.190163 864 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 15:01:33.190373 864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 15:01:33.190395 864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 15:01:33.190500 864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 15:01:33.194832 864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 15:01:33.194903 864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 15:01:33.194931 864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 15:01:33.194968 864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 15:01:33.194984 864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 15:01:33.199618 864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.71.148:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.71.190:43209,127.10.71.189:45787,127.10.71.188:37099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/krb5kdc/kudu_127.10.71.188.keytab
--principal=kudu/127.10.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.71.188:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 15:01:33.200430 864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 15:01:33.201203 864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 15:01:33.208220 871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:33.208251 870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 15:01:33.208251 873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 15:01:33.208561 864 server_base.cc:1034] running on GCE node
I20260619 15:01:33.209057 864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 15:01:33.209967 864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 15:01:33.211169 864 hybrid_clock.cc:648] HybridClock initialized: now 1781881293211139 us; error 37 us; skew 500 ppm
I20260619 15:01:33.212232 858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } attempt: 2
I20260619 15:01:33.212831 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.212614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49804)
0619 15:01:33.212727 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.212777 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 15:01:33.213483 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.213284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49818)
0619 15:01:33.213378 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.213426 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 15:01:33.213593 858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:37099: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
I20260619 15:01:33.213872 864 init.cc:377] Logged in from keytab as kudu/127.10.71.188@KRBTEST.COM (short username kudu)
I20260619 15:01:33.214696 864 webserver.cc:460] Webserver started at http://127.10.71.188:44049/ using document root <none> and password file <none>
I20260619 15:01:33.215206 864 fs_manager.cc:359] Metadata directory not provided
I20260619 15:01:33.215263 864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 15:01:33.215464 864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 15:01:33.217341 864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data/instance:
uuid: "21e652c2e4484bdb94a7093ff62bab1f"
format_stamp: "Formatted at 2026-06-19 15:01:33 on dist-test-slave-3qdh"
server_key: "c7d87e3e564aad69f530c5d090c66375"
server_key_iv: "93220211c37b23e6b7c2dec5abd219ec"
server_key_version: "encryptionkey@0"
I20260619 15:01:33.218045 864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal/instance:
uuid: "21e652c2e4484bdb94a7093ff62bab1f"
format_stamp: "Formatted at 2026-06-19 15:01:33 on dist-test-slave-3qdh"
server_key: "c7d87e3e564aad69f530c5d090c66375"
server_key_iv: "93220211c37b23e6b7c2dec5abd219ec"
server_key_version: "encryptionkey@0"
I20260619 15:01:33.221208 864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 15:01:33.223130 880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 15:01:33.224268 864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 15:01:33.224411 864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data,/tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal
uuid: "21e652c2e4484bdb94a7093ff62bab1f"
format_stamp: "Formatted at 2026-06-19 15:01:33 on dist-test-slave-3qdh"
server_key: "c7d87e3e564aad69f530c5d090c66375"
server_key_iv: "93220211c37b23e6b7c2dec5abd219ec"
server_key_version: "encryptionkey@0"
I20260619 15:01:33.224490 864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 15:01:33.226717 783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } attempt: 2
I20260619 15:01:33.227283 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.227061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49832)
0619 15:01:33.227172 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.227230 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 15:01:33.227993 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.227803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49836)
0619 15:01:33.227895 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.227942 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 15:01:33.228180 783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.71.188:37099: Network error: Client connection negotiation failed: client connection to 127.10.71.188:37099: connect: Connection refused (error 111)
I20260619 15:01:33.249614 864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 15:01:33.251747 864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 15:01:33.251909 864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 15:01:33.257592 864 rpc_server.cc:307] RPC server started. Bound to: 127.10.71.188:37099
I20260619 15:01:33.257817 932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.71.188:37099 every 8 connection(s)
I20260619 15:01:33.258325 864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/data/info.pb
I20260619 15:01:33.260519 933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 15:01:33.263343 933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:33.264559 10526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskn16cqy/build/debug/bin/kudu as pid 864
I20260619 15:01:33.264699 10526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskn16cqy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781881028756249-10526-0/minicluster-data/master-2/wal/instance
I20260619 15:01:33.265004 10526 external_mini_cluster.cc:1428] Setting key edf254147c608743df1aeffabaec495f
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 15:01:33.276058 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.264905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.10.71.190:43209)
0619 15:01:33.265062 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.265066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.265834 (+ 768us) server_negotiation.cc:407] Connection header received
0619 15:01:33.266702 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.266705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.266746 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.266837 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.267758 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.268520 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.269708 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.269996 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.270085 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.272239 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.272242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.272246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.272275 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.273934 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.274496 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.274499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.274500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.274569 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.274940 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.274943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.274945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.275207 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.275274 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.275792 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.275895 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 15:01:33.276948 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.265540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.10.71.188:37099)
0619 15:01:33.265844 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.265853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.265966 (+ 113us) server_negotiation.cc:407] Connection header received
0619 15:01:33.266546 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.266561 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.266760 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.266913 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.267689 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.268788 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.269808 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.270235 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.270301 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.272705 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.272714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.272726 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.272753 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.274715 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.275166 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.275170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.275171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.275266 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.275497 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.275500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.275502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.275724 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.275885 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.276083 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.276210 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":1}
I20260619 15:01:33.276948 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.265316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.190:43209 (local address 127.0.0.1:46102)
0619 15:01:33.265705 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.265750 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.266537 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.266906 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.266911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.267325 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.267612 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.267630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.268626 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.268629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.269601 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.269611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.270115 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.270117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.270230 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.270680 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.270693 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.272053 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.274055 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.274064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.274076 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.274390 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.274658 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.274661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.274663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.274837 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.275293 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.275300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.275560 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.275731 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.276003 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"spinlock_wait_cycles":2432,"thread_start_us":104,"threads_started":1}
I20260619 15:01:33.277946 933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.190@KRBTEST.COM
I20260619 15:01:33.285693 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.278335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.189:45787 (local address 127.0.0.1:58264)
0619 15:01:33.278627 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.278640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.278709 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.278916 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.278918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.279064 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.279131 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.279136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.279774 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.279777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.280228 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.280235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.280668 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.280671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.280753 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.281154 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.281164 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.282452 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.284284 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.284287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.284289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.284551 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.284810 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.284815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.284818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.284922 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.285257 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.285260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.285342 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.285459 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.285564 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260619 15:01:33.285717 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.278512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.10.71.189:45787)
0619 15:01:33.278617 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.278620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.278677 (+ 57us) server_negotiation.cc:407] Connection header received
0619 15:01:33.278782 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.278784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.278822 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.278906 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.279234 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.279671 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.280352 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.280569 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.280668 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.282569 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.282572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.282574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.282595 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.284165 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.284644 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.284647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.284648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.284715 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.285023 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.285025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.285026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.285159 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.285264 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.285508 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.285599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 15:01:33.286285 933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
I20260619 15:01:33.286878 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.278755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.10.71.190:43209)
0619 15:01:33.278845 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.278847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.278986 (+ 139us) server_negotiation.cc:407] Connection header received
0619 15:01:33.279071 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.279073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.279111 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.279182 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.279683 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.280097 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.280818 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.281186 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.281302 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.283418 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.283421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.283424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.283450 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.285045 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.285394 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.285398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.285400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.285478 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.285914 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.285918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.285920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.286112 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.286329 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.286727 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.286798 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.188@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.71.189@KRBTEST.COM
I20260619 15:01:33.294064 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.286609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49852)
0619 15:01:33.286818 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.286828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.286892 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.287290 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.287293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.287443 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.287511 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.287516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.288534 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.288540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.288997 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.289003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.289436 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.289439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.289529 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.289985 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.289998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.291060 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.292682 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.292688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.292691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.292864 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.293141 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.293144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.293146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.293220 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.293548 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.293550 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.293623 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.293768 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.293888 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1}
I20260619 15:01:33.294148 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.286744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.10.71.188:37099)
0619 15:01:33.286820 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.286825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.286861 (+ 36us) server_negotiation.cc:407] Connection header received
0619 15:01:33.287161 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.287163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.287192 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.287247 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.287619 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.288444 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.289100 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.289331 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.289431 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.291174 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.291177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.291179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.291203 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.292583 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.292972 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.292976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.292978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.293044 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.293309 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.293312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.293314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.293459 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.293556 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.293807 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.293939 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":21}
I20260619 15:01:33.295712 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.287791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.10.71.189:45787)
0619 15:01:33.287874 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.287877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.287890 (+ 13us) server_negotiation.cc:407] Connection header received
0619 15:01:33.287976 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.287979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.288014 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.288086 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.288678 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.289114 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.289949 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.290277 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.290386 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.292748 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.292751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.292752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.292771 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.294106 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.294611 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.294615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.294617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.294698 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.294960 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.294965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.294966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.295122 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.295339 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.295456 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.295563 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 15:01:33.296916 933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: Bootstrap starting.
I20260619 15:01:33.299166 10526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 15:01:33.299432 933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:33.300222 933 log.cc:824] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:33.300348 757 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3642be3a1ae3484988cbc9a6fe6570ac"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46112
I20260619 15:01:33.301823 933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: No bootstrap required, opened a new log
I20260619 15:01:33.304237 933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.304471 933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e652c2e4484bdb94a7093ff62bab1f, State: Initialized, Role: FOLLOWER
I20260619 15:01:33.304723 858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } attempt: 3
I20260619 15:01:33.304764 933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.305349 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.305470 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.306033 933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.189@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
W20260619 15:01:33.310540 950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:33.310603 950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:33.313171 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.305069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49858)
0619 15:01:33.305247 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.305261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.305352 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.305638 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.305640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.305792 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.305854 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.305859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.306597 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.306602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.307304 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.307313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.307780 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.307783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.307859 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.308351 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.308364 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.309645 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.311676 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.311679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.311681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.311901 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.312204 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.312209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.312212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.312315 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.312742 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.312746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.312826 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.312918 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.313034 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 15:01:33.313226 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.305140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.10.71.188:37099)
0619 15:01:33.305354 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.305358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.305373 (+ 15us) server_negotiation.cc:407] Connection header received
0619 15:01:33.305437 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.305440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.305500 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.305604 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.305965 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.306488 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.307406 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.307679 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.307778 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.309798 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.309804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.309806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.309831 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.311575 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.312003 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.312007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.312008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.312095 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.312415 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.312418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.312420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.312636 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.312733 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.312994 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.313091 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 15:01:33.309837 864 master_runner.cc:186] Error getting master registration for 127.10.71.190:43209: OK, instance_id { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" instance_seqno: 1781881292868356 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:01:33.314034 864 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:01:33.315464 858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: Bootstrap starting.
I20260619 15:01:33.317323 858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:33.317938 858 log.cc:824] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:33.319641 858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: No bootstrap required, opened a new log
I20260619 15:01:33.323032 858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.323203 858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801ee8abb07548bc8cbe0226360c0cce, State: Initialized, Role: FOLLOWER
I20260619 15:01:33.323478 858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.323873 953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.323992 953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.324589 858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:33.327265 783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } attempt: 3
W20260619 15:01:33.329123 964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:33.329185 964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:33.327526 790 master_runner.cc:186] Error getting master registration for 127.10.71.190:43209: OK, instance_id { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" instance_seqno: 1781881292868356 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 15:01:33.330652 790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 15:01:33 dist-test-slave-3qdh krb5kdc[702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781881292, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.71.190@KRBTEST.COM for kudu/127.10.71.188@KRBTEST.COM
I20260619 15:01:33.336158 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.327588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.71.188:37099 (local address 127.0.0.1:49872)
0619 15:01:33.327712 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 15:01:33.327727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 15:01:33.327814 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 15:01:33.328367 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 15:01:33.328370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 15:01:33.328555 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 15:01:33.328680 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.328688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.329578 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.329584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.330276 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 15:01:33.330286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.330810 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.330815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 15:01:33.330903 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.331350 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 15:01:33.331365 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 15:01:33.332728 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 15:01:33.334600 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.334603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.334606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.334865 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.335206 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 15:01:33.335208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 15:01:33.335210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 15:01:33.335307 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 15:01:33.335680 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 15:01:33.335683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 15:01:33.335783 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 15:01:33.335889 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 15:01:33.336006 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 15:01:33.336174 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.327869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.10.71.188:37099)
0619 15:01:33.328146 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.328151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.328165 (+ 14us) server_negotiation.cc:407] Connection header received
0619 15:01:33.328199 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.328202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.328241 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.328344 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.328794 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.329465 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.330405 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.330694 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.330795 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.332853 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.332856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.332859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.332881 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.334499 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.334975 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.334979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.334981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.335082 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.335400 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.335403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.335405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.335592 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.335682 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.335951 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.336050 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260619 15:01:33.338970 783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: Bootstrap starting.
I20260619 15:01:33.341144 783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: Neither blocks nor log segments found. Creating new log.
I20260619 15:01:33.341874 783 log.cc:824] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: Log is configured to *not* fsync() on all Append() calls
I20260619 15:01:33.344230 783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: No bootstrap required, opened a new log
I20260619 15:01:33.348387 783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.348675 783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3642be3a1ae3484988cbc9a6fe6570ac, State: Initialized, Role: FOLLOWER
I20260619 15:01:33.349149 783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.349681 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.349858 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.350852 757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 15:01:33.350965 757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:33.352075 757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.352355 783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 15:01:33.353036 757 leader_election.cc:288] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [CANDIDATE]: Term 1 election: Requested vote from peers 801ee8abb07548bc8cbe0226360c0cce (127.10.71.189:45787), 21e652c2e4484bdb94a7093ff62bab1f (127.10.71.188:37099)
I20260619 15:01:33.354058 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "801ee8abb07548bc8cbe0226360c0cce"
I20260619 15:01:33.354255 832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [term 0 FOLLOWER]: Advancing to term 1
I20260619 15:01:33.355535 832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3642be3a1ae3484988cbc9a6fe6570ac in term 1.
I20260619 15:01:33.355918 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21e652c2e4484bdb94a7093ff62bab1f"
I20260619 15:01:33.356140 907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [term 0 FOLLOWER]: Advancing to term 1
W20260619 15:01:33.357745 978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 15:01:33.357910 978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 15:01:33.357913 907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3642be3a1ae3484988cbc9a6fe6570ac in term 1.
I20260619 15:01:33.359907 734 leader_election.cc:302] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3642be3a1ae3484988cbc9a6fe6570ac, 801ee8abb07548bc8cbe0226360c0cce; no voters:
I20260619 15:01:33.360333 967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 15:01:33.360529 967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [term 1 LEADER]: Becoming Leader. State: Replica: 3642be3a1ae3484988cbc9a6fe6570ac, State: Running, Role: LEADER
I20260619 15:01:33.360800 967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } }
I20260619 15:01:33.362227 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3642be3a1ae3484988cbc9a6fe6570ac. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.362321 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: This master's current role is: LEADER
W20260619 15:01:33.365581 736 tablet.cc:2369] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 15:01:33.366439 715 master_runner.cc:263] No leader master found from master 3642be3a1ae3484988cbc9a6fe6570ac
I20260619 15:01:33.366500 715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 15:01:33.366580 988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 15:01:33.367326 988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 15:01:33.369969 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.359748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.10.71.189:45787)
0619 15:01:33.359839 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.359843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.361056 (+ 1213us) server_negotiation.cc:407] Connection header received
0619 15:01:33.361207 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.361210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.361248 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.361290 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.362205 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.362851 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.364033 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.364370 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.364441 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.366659 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.366662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.366665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.366686 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.368288 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.368794 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.368798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.368800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.368877 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.369159 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.369162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.369164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.369349 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.369450 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.369749 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.369842 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 15:01:33.379092 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.360379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.10.71.188:37099)
0619 15:01:33.360500 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.360503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.371286 (+ 10783us) server_negotiation.cc:407] Connection header received
0619 15:01:33.371416 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.371419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.371457 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.371526 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.372357 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.373062 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.374067 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.374397 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.374465 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.376011 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.376014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.376017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.376038 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.377606 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.378006 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.378009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.378011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.378082 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.378330 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.378333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.378336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.378522 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.378619 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.378887 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.378973 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 15:01:33.380340 907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [term 1 FOLLOWER]: Refusing update from remote peer 3642be3a1ae3484988cbc9a6fe6570ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:33.381004 985 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:33.384091 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3642be3a1ae3484988cbc9a6fe6570ac. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.384160 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:33.365329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.10.71.190:43209)
0619 15:01:33.365420 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.365424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.376528 (+ 11104us) server_negotiation.cc:407] Connection header received
0619 15:01:33.376697 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.376702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.376753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.376803 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.377592 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.378246 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.379175 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:33.379457 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:33.379528 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:33.381017 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 15:01:33.381020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 15:01:33.381022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 15:01:33.381044 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 15:01:33.382484 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.382880 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.382883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.382884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.382950 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 15:01:33.383182 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 15:01:33.383185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 15:01:33.383186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 15:01:33.383344 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 15:01:33.383433 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:33.383922 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 15:01:33.384024 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 15:01:33.384203 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.385421 832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [term 1 FOLLOWER]: Refusing update from remote peer 3642be3a1ae3484988cbc9a6fe6570ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 15:01:33.385896 985 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 15:01:33.389153 953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3642be3a1ae3484988cbc9a6fe6570ac. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.389253 953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.399226 985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.399335 985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:33.399569 990 mvcc.cc:205] Tried to move back new op lower bound from 7298585777674842112 to 7298585777608691712. Current Snapshot: MvccSnapshot[applied={T|T < 7298585777674842112}]
I20260619 15:01:33.400321 953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.400424 953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.400952 993 mvcc.cc:205] Tried to move back new op lower bound from 7298585777674842112 to 7298585777608691712. Current Snapshot: MvccSnapshot[applied={T|T < 7298585777674842112}]
I20260619 15:01:33.401543 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.401644 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac [sys.catalog]: This master's current role is: LEADER
I20260619 15:01:33.402544 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3642be3a1ae3484988cbc9a6fe6570ac" member_type: VOTER last_known_addr { host: "127.10.71.190" port: 43209 } } peers { permanent_uuid: "801ee8abb07548bc8cbe0226360c0cce" member_type: VOTER last_known_addr { host: "127.10.71.189" port: 45787 } } peers { permanent_uuid: "21e652c2e4484bdb94a7093ff62bab1f" member_type: VOTER last_known_addr { host: "127.10.71.188" port: 37099 } } }
I20260619 15:01:33.402657 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 15:01:33.403131 991 mvcc.cc:205] Tried to move back new op lower bound from 7298585777674842112 to 7298585777608691712. Current Snapshot: MvccSnapshot[applied={T|T < 7298585777674842112}]
I20260619 15:01:33.413507 988 catalog_manager.cc:1347] Generated new cluster ID: 41d2e2b3d9864e54bdc7bc8f6a349ca2
I20260619 15:01:33.413563 988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 15:01:33.425647 988 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 15:01:33.427382 988 catalog_manager.cc:1504] Loading token signing keys...
I20260619 15:01:33.438687 988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3642be3a1ae3484988cbc9a6fe6570ac: Generated new TSK 0
I20260619 15:01:33.439291 988 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 15:01:33.469022 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.466019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.10.71.189:45787)
0619 15:01:33.466125 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.466128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.467816 (+ 1688us) server_negotiation.cc:407] Connection header received
0619 15:01:33.467988 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.467991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.468036 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.468082 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.468892 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58284: BlockingRecv error: recv got EOF from 127.0.0.1:58284 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 15:01:33.470112 936 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.466637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.10.71.188:37099)
0619 15:01:33.466732 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.466735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.468506 (+ 1771us) server_negotiation.cc:407] Connection header received
0619 15:01:33.469437 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.469440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.469496 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.469538 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.470009 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49896: BlockingRecv error: recv got EOF from 127.0.0.1:49896 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 15:01:33.471724 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.467260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.10.71.190:43209)
0619 15:01:33.467352 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.467356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.470647 (+ 3291us) server_negotiation.cc:407] Connection header received
0619 15:01:33.470993 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.470996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.471041 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.471097 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.471607 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46120: BlockingRecv error: recv got EOF from 127.0.0.1:46120 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 15:01:33.475854 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.474591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.10.71.189:45787)
0619 15:01:33.474704 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.474708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.474966 (+ 258us) server_negotiation.cc:407] Connection header received
0619 15:01:33.475182 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.475185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.475249 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.475297 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.475775 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58290: BlockingRecv error: recv got EOF from 127.0.0.1:58290 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 15:01:33.480477 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.479198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.10.71.189:45787)
0619 15:01:33.479276 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.479280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.479727 (+ 447us) server_negotiation.cc:407] Connection header received
0619 15:01:33.479876 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.479879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.479917 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.479958 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.480398 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58296: BlockingRecv error: recv got EOF from 127.0.0.1:58296 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 15:01:33.485617 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.484166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.10.71.189:45787)
0619 15:01:33.484256 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.484259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.484812 (+ 553us) server_negotiation.cc:407] Connection header received
0619 15:01:33.484990 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.484993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.485037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.485084 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.485540 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58308: BlockingRecv error: recv got EOF from 127.0.0.1:58308 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 15:01:33.492434 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.491236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.10.71.189:45787)
0619 15:01:33.491331 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.491334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.491751 (+ 417us) server_negotiation.cc:407] Connection header received
0619 15:01:33.491888 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.491890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.491925 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.491963 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.492363 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58320: BlockingRecv error: recv got EOF from 127.0.0.1:58320 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 15:01:33.503011 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.501917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.10.71.189:45787)
0619 15:01:33.501990 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.501994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.502373 (+ 379us) server_negotiation.cc:407] Connection header received
0619 15:01:33.502518 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.502520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.502554 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.502590 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.502943 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58326: BlockingRecv error: recv got EOF from 127.0.0.1:58326 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 15:01:33.521740 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.520750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.10.71.189:45787)
0619 15:01:33.520854 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.520860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.521060 (+ 200us) server_negotiation.cc:407] Connection header received
0619 15:01:33.521164 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.521167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.521210 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.521283 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.521644 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58338: BlockingRecv error: recv got EOF from 127.0.0.1:58338 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 15:01:33.556571 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.555545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.10.71.189:45787)
0619 15:01:33.555664 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.555669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.555840 (+ 171us) server_negotiation.cc:407] Connection header received
0619 15:01:33.555962 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.555964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.556011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.556063 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.556490 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58346: BlockingRecv error: recv got EOF from 127.0.0.1:58346 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 15:01:33.623144 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.622293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.10.71.189:45787)
0619 15:01:33.622395 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.622398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.622514 (+ 116us) server_negotiation.cc:407] Connection header received
0619 15:01:33.622612 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.622614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.622652 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.622717 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.623061 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58362: BlockingRecv error: recv got EOF from 127.0.0.1:58362 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 15:01:33.754153 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:33.753052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.10.71.189:45787)
0619 15:01:33.753185 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:33.753189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:33.753328 (+ 139us) server_negotiation.cc:407] Connection header received
0619 15:01:33.753414 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:33.753416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:33.753459 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:33.753524 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:33.754034 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58372: BlockingRecv error: recv got EOF from 127.0.0.1:58372 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 15:01:34.013067 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 15:01:34.012082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.10.71.189:45787)
0619 15:01:34.012237 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:34.012241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:34.012257 (+ 16us) server_negotiation.cc:407] Connection header received
0619 15:01:34.012427 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:34.012429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:34.012478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:34.012550 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 15:01:34.012939 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58378: BlockingRecv error: recv got EOF from 127.0.0.1:58378 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 15:01:34.311683 950 catalog_manager.cc:1259] Loaded cluster ID: 41d2e2b3d9864e54bdc7bc8f6a349ca2
I20260619 15:01:34.311753 950 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: loading cluster ID for follower catalog manager: success
I20260619 15:01:34.313320 950 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: acquiring CA information for follower catalog manager: success
I20260619 15:01:34.314365 950 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21e652c2e4484bdb94a7093ff62bab1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:01:34.330229 964 catalog_manager.cc:1259] Loaded cluster ID: 41d2e2b3d9864e54bdc7bc8f6a349ca2
I20260619 15:01:34.330286 964 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: loading cluster ID for follower catalog manager: success
I20260619 15:01:34.331612 964 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: acquiring CA information for follower catalog manager: success
I20260619 15:01:34.332496 964 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 801ee8abb07548bc8cbe0226360c0cce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 15:01:34.530928 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:34.527037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.10.71.189:45787)
0619 15:01:34.527262 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:34.527265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:34.527385 (+ 120us) server_negotiation.cc:407] Connection header received
0619 15:01:34.527505 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:34.527507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:34.527623 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:34.527714 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:34.528017 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.528597 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.529508 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.529703 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.529765 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:34.530039 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:34.530485 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:34.530574 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:34.530655 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 15:01:34.530696 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260619 15:01:34.536907 1068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:34.533708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.10.71.188:37099)
0619 15:01:34.533925 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:34.533929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:34.533947 (+ 18us) server_negotiation.cc:407] Connection header received
0619 15:01:34.534018 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:34.534020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:34.534117 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:34.534212 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:34.534382 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.534867 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.535624 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.535852 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.535936 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:34.536160 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:34.536576 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:34.536669 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:34.536756 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 15:01:34.536801 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
I20260619 15:01:34.543617 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:34.539510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.10.71.189:45787)
0619 15:01:34.539707 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:34.539714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:34.539905 (+ 191us) server_negotiation.cc:407] Connection header received
0619 15:01:34.540075 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:34.540078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:34.540123 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:34.540171 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:34.540613 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.541362 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.542389 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.542624 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.542712 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:34.543180 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:34.543258 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:34.543324 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:34.543434 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 15:01:34.543478 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260619 15:01:34.543779 1068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:34.540023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.10.71.188:37099)
0619 15:01:34.540120 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:34.540123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:34.540369 (+ 246us) server_negotiation.cc:407] Connection header received
0619 15:01:34.540547 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:34.540549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:34.540611 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:34.540655 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:34.540997 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.541644 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.542641 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.542907 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.543028 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:34.543304 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:34.543372 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:34.543440 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:34.543588 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 15:01:34.543644 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 15:01:34.544144 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 15:01:34.539514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.10.71.190:43209)
0619 15:01:34.539751 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 15:01:34.539758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 15:01:34.539775 (+ 17us) server_negotiation.cc:407] Connection header received
0619 15:01:34.539966 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 15:01:34.539970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 15:01:34.540114 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 15:01:34.540233 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 15:01:34.540525 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.541066 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.542193 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 15:01:34.542550 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 15:01:34.542646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 15:01:34.543271 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 15:01:34.543570 (+ 299us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 15:01:34.543644 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 15:01:34.543928 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 15:01:34.544003 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260619 15:01:34.546804 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 715
I20260619 15:01:34.553049 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 790
I20260619 15:01:34.558429 10526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskn16cqy/build/debug/bin/kudu with pid 864
2026-06-19T15:01:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2900 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5769 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (265799 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST